DEV Community: Johannes Dienst

Use Playwright and AskUI Together

Johannes Dienst — Wed, 25 Sep 2024 08:42:53 +0000

Use Playwright and AskUI Together

Playwright is the de facto standard when it comes to reliably testing browser applications across browsers and operating systems.

But what if your workflow under test leaves your browser and requires you to automate on a desktop application?

AskUI is not confined to a specific application because it uses visual selectors and real human user-like input with keypresses and natural mouse movement. This makes it an ideal tool to accompany Playwright.

In this post you will learn to setup Playwright together with AskUI to define workflows that can leave the browser.

Prerequisites

AskUI installed and configured on your system (Windows, Linux, macOS)

Install Playwright

Playwright for JavaScript/TypeScript consists of two components.

The playwright node package
A browser installed through Playwright library

First you have to install playwright as a dev-dependency. Execute the following command in your terminal:

npm install --save-dev playwright

For this blog you will use Chromium as a browser. The installation through playwright is done like this:

npx playwright install --with-deps chromium

Configure Playwright

You will configure Playwright and the Chromium browser in the file helpers/askui-helper.ts, so it will work correctly together with Jest.

import { UiControlClient, UiController } from 'askui';
import { AskUIAllureStepReporter } from '@askui/askui-reporters';

// Server for controlling the operating system
let uiController: UiController;

// Client is necessary to use the askui API
let aui: UiControlClient;

// 1. Playwright setup
import { chromium, devices, Browser, BrowserContext } from 'playwright';
let pwBrowser: Browser;
let pwContext: BrowserContext;

jest.setTimeout(60 * 1000 * 60);

beforeAll(async () => {

  // 2. Playwright setup
  pwBrowser = await chromium.launch({ headless: false, slowMo: 100 });
  pwContext = await pwBrowser.newContext(devices['Desktop Chrome']);

  uiController = new UiController({
    /**
      * Select the display you want to run your tests on, display 0 is your main display;
      * ignore if you have only one display
      */
    display: 0,
  });

  await uiController.start();

  aui = await UiControlClient.build({
    credentials: {
      workspaceId: '<your-workspace-id>',
      token: '<your-access-token>',
    },
    reporter: new AskUIAllureStepReporter(),
  });

  await aui.connect();
});

...

afterAll(async () => {

  // 3. Playwright setup
  await pwContext.close();
  await pwBrowser.close();

  aui.disconnect();

  await uiController.stop();
});

export { aui };

// 4. Playwright setup
export { pwContext };

Import the necessary classes from playwright and define the variables
Configure chromium to be not headless and also slow down its execution. Also we want a Desktop Chrome as we are on desktop.
Close the BrowserContext and the Browser after the execution finished.
Export the Context so you can use it in your workflow

Write a Workflow

With the BrowserContext set up, hop over to the file my-first-askui-test-seuite.test.ts and import it there. Add this to the start of the file:

import { pwContext } from './helpers/askui-helper';

Now you can implement a simple workflow:

Open a new tab
Navigate to https://google.com/
Click away the cookie banner
Fill the search field with Playwright framework
Press Enter on keyboard
Switch to image search by clicking on the Button architecture
Move the mouse cursor to the element that has an image with a red and a green mask in it (Playwright logo description)

describe('jest with askui', () => {
  it('should open askui.com in Chrome with playwright', async () => {
    let page = await pwContext.newPage();
    await page.goto('https://google.com/');
    await page.getByRole('button', { name: 'Reject all' }).click();
    await page.locator('//textarea[@name="q"]').fill('Playwright framework');
    await page.keyboard.press('Enter');
    await page.waitForTimeout(1000);

    // AskUI workflow
    // Make sure you have the focus on the browser
    await aui.mouseLeftClick().exec();
    await aui.click().button().withText('Images').exec();
    await aui.click().text('architecture').exec();
    await aui.moveMouseTo().element().matching('image with a red and a green mask in it').exec()

    await page.waitForTimeout(5000);
  });
});

Testrun

To run this you need to run this command in your terminal:

// Linux + macOS
npm run askui

// Windows
AskUI-RunProject

The following gif shows the execution of the whole workflow:

Conclusion

Combining AskUI with Playwright enhances automation capabilities to extend beyond web browsers. While Playwright is a solid choice for browser automation, AskUI takes the reins when dealing with desktop apps and operating systems.

This integration offers a practical toolkit for workflows across various applications for efficient automation.

Run AskUI Workflows with Pipedream for Smoke Testing

Johannes Dienst — Wed, 07 Aug 2024 14:18:25 +0000

Dylan Pierce shared an interesting approach to smoke testing at our meetup: Fill out a contact form with the help of Computer Vision & Large Language Models. Especially the vision part was impressive but it lacked one specific feature: Interaction with the form. How can you be sure your form works if you do not try to fill it out and send it?

This is where I thought an AskUI integration into the Pipedream workflow could be useful. AskUI uses visual selectors instead of code selectors and can interact like a human with a form like that. Giving us a real smoke test!

In this blog I will describe how I integrated AskUI into a Pipedream workflow, so we get the benefit of visual selection and user interaction.

Prerequisites

AskUI Controller installed and configured on a remote accessible system like Gitpod or a cloud machine. You can use our Try-Out-Repository or install on your own system (Windows, Linux, macOS).
Pipedream account and a workflow. Follow their Introduction

What Will We Build? Aka the Use Case

Dylan Pierce showed an awesome use case with Pipedream and Puppeteer, where he implemented a smoke test with the help of AI without writing any selectors by himself. I highly recommend to watch the recording: https://youtu.be/o6hej9Ip2vs

Our Use Case: Smoke Test with Visual Selectors

Dylans use case involved querying a Large Language/Multimodal Model to implement the smoke test. We will modify this a little bit to use the visual selectors from AskUI, which do not rely on the specific UI-technology but identify elements through their appearance with an AI vision model.

Here are the steps we will implement:

Trigger the smoke test once per day
Perform the smoke test with AskUI on a remote system
Send an email if the smoke test was successful or not

1. Add a Trigger

The first thing Pipedream wants us to add is a trigger. We add a Schedule trigger that will run our Workflow everyday at 9:00 AM.

2. Prepare a Custom Code Action

AskUI does not have an action in Pipedream. So we will make use of the AskUI node package and run a custom code action. For this we a a _Code action. In this action we will fill out the simple authentication from https://authenticationtest.com/simpleFormAuth/. If we see the success page we will send

We have to do the following steps:

Add a property uiControllerUrl, so we do not have to hardcode it into the code
Store our credentials in environment variables
Import UiControlClient from the askui node package
Configure the AskUI UiControlClient to use the credentials and the uiControllerUrl

The first thing we want to add to our custom code is the uiControllerUrl. Click Refresh fields so the Configure tab shows up at the start of the action. Here is the relevant code snippet.

...
export default defineComponent({

  props: {
    uiControllerUrl: { type: "string" }
  },

  async run({ steps, $ }) {
...

Then head over to environment variables and add your workspaceId and accessToken there. You obtained those by following the prerequisites instructions above. You can now setup the connection to the AskUI Controller over the UiControlClient like this. Notice how easy it is to use arbitrary node-packages in Pipedream? I only needed to import UiControlClient and it just worked 🥳.

Note: The following code also contains the teardown.

import { UiControlClient } from 'askui';

...
  async run({ steps, $ }) {
    const result = {};

    const aui = await UiControlClient.build({
      credentials: {
        workspaceId: process.env.workspaceId,
        token: process.env.token,
      },
      uiControllerUrl: this.uiControllerUrl
    });
    await aui.connect();

    // AskUI Workflow will be added here

    aui.disconnect();

    return result;
  },
})

3. Write the AskUI Workflow

When you look at our example form, we want to fill out, you notice that we have to do the following things:

Write simpleForm@authenticationtest.com into the textfield E-Mail Address
Write pa$$w0rd into the next textfield
Click the button Log In
Validate the success

The less inference we invoke the faster the AskUI workflow will execute. Everything that prompts AskUI to search for an element on the screen invokes inference. So let us try to invoke the inference only once by finding the first textfield for the E-Mail Address. Then we will use keypresses to navigate the form. This is the code to achieve this:

// This line only works with the Gitpod setup shown in the next section
// Select the browser url textfield for your use case with the appropriate instruction!
await aui.typeIn('https://authenticationtest.com/simpleFormAuth/')
         .textfield()
         .contains()
         .text()
         .containsText('docs.askui')
         .exec();
await aui.pressKey('enter').exec();

// Fill out the form
await aui.typeIn('simpleForm@authenticationtest.com')
         .textfield()
         .contains()
         .text('E-Mail Address')
         .exec();
await aui.pressKey('tab').exec();
await aui.type('pa$$w0rd').exec();
await aui.pressKey('tab').exec();
await aui.pressKey('enter').exec();

// Check if the the login succeeded: Login Success is shown on the page
// Pass result to next step
try {
  await aui.expect().text('Login Success').exists().exec();
  result.success = "Smoke Test successful!";
} catch(error) {
  result.success = "Smoke Test failed!";
}

aui.disconnect();

return result;

4. Send an Email

Doing a smoke test without reporting about its success state would not help us. So we will just send us an email with the Send Yourself an Email action.

As subject we choose Smoke Test State and for the text we reference our success variable we returned in our action above with {{steps.code.$return_value.success}}.

Gitpod As Remote Machine

If you do not have a remote machine ready-to-go you can use a service like Gitpod. We have a prepared Try-Out-Repository which comes with AskUI already setup and a VNC to observe the AskUI workflow. Start the repository in Gitpod over the Open in Gitpod-button and let it finish the predefined AskUI workflow. When it reached the AskUI Docs (docs.askui.com) maximize the browser window in the Simple Browser tab.

Switch to the TERMINAL tab and start the AskUI-Controller with the following command:

./node_modules/askui/dist/release/latest/linux/askui-ui-controller.AppImage

Also make sure that you expose the port to the AskUI Controller (open lock icon). Head to the PORTS tab and make the port 6769 public. Then copy the URL and add it as the property uiControllerUrl in the Pipedream action.

Conclusion

Building a smoke test with Pipedream and AskUI was a practical use case to see how both tools integrate. The simplicity of Pipedream and its ability to integrate JavaScript code and Node packages was helpful. With that AskUI could be setup seamlessly inside an action and connected to an external AskUI Controller.

Use AskUI and Cucumber Together

Johannes Dienst — Wed, 24 Jul 2024 07:27:45 +0000

Use AskUI and Cucumber Together

By defining the behavior of a system in a structured format like Gherkin, Behavior-Driven Development (BDD) enables teams to bridge the gap between stakeholders, testers, and developers, avoiding misunderstandings and reducing rework. As a collaborative approach, BDD encourages all parties to work together from the outset, ensuring that everyone is on
the same page and that requirements are accurately captured.

In this process, Cucumber is a popular tool used to implement BDD, enabling teams to write clear, executable tests that ensure the system behaves as expected.

In this blog post, we'll show you how to set up Cucumber in conjunction with AskUI to define AskUI workflows using BDD principles.

Prerequisites

AskUI installed and configured on your system (Windows, Linux, macOS)
Delete askui_example/my-first-askui-test-suite.test.ts after initialization

Prepare Setup

Cucumber does not play nice with AskUI's default setup yet (Version 0.20.3). For AskUI to play nice with Cucumber you need to do two small preparations as AskUI uses Jest as its runner.

1. Change Jest's `testEnvironmentOptions`

In the file askui_example/helpers/jest.config.ts you have to disable that code is included in the run report. You achieve this by adding a testEnvironmentOptions property with the addCodeInReport property set to false.

const config: Config.InitialOptions = {
  ...
  testEnvironment: '@askui/jest-allure-circus',
  testEnvironmentOptions: {
    addCodeInReport: false
  },
};
...

2. Tell Jest Where to Find the Implementation for The Step Definitions

Also in askui_example/helpers/jest.config.ts you need to expand the default testMatch property. It must include files ending in step.ts because we will store the implementation there.

...
const config: Config.InitialOptions = {
  ...
  testEnvironment: '@askui/jest-allure-circus',
  testEnvironmentOptions: {
    addCodeInReport: false
  },
  testMatch: [ "**/__tests__/**/*.[jt]s?(x)", "**/?(*.)+(spec|test|step).[jt]s?(x)" ]
};
...

Install `jest-cucumber`

The easiest way to use Jest together with Cucumber is the npm-package jest-cucumber. Let's install it with the following command:

npm install --save-dev jest-cucumber

Create a Basic Features File

Create a folder features and in there a Feature file NavigateToWebsite.feature

project_root/
├─ askui_example/
├─ features/
  ├─ NavigateToWebsite.feature
├─ node_modules/
├─ ...

Write the following basic Feature into this file:

Feature: Navigate to a website

Scenario: Entering the correct URL into the browser address bar
  Given I am on the Google search page
  When I type in the URL for AskUI practice page
  Then I will land on the webpage

Create the Step Definitions Implementations

Create the step definition file askui_example/navigate-to-url.step.ts where each test maps to a specific scenario.

import { defineFeature, loadFeature } from 'jest-cucumber';
import { aui } from './helpers/askui-helper';

// Load the feature file
const feature = loadFeature('features/NavigateToWebsite.feature');

defineFeature(feature, test => {

  // Maps to 'Scenario' in your feature file
  test('Entering the correct URL into the browser address bar', ({ given, when, then }) => {

    given('I am on the Google search page', async () => {
      await aui.moveMouse(500, 500).exec();
      await aui.mouseLeftClick().exec();
      await aui.pressTwoKeys('command', 't').exec();
    });

    when('I type in the URL for AskUI practice page', async () => {
      await aui.typeIn('https://askui.github.io/askui-practice-page/').textfield().exec();
      await aui.pressKey('enter').exec();
    });

    then('I will land on the webpage', async () => {
      await aui.expect().text('Welcome to the AskUI Practice Page').exists().exec();
    });

  });
});

Run The Workflow

Open your browser in full screen and start the workflow with:

npm run askui

You should see that the workflow run will open a new tab and navigate to AskUI's practice page.

Conclusion

Combining AskUI with Cucumber enables you to write AskUI workflows in BDD style. Executing your tests like a real human-user will make them more realistic for every stakeholder.

Common Terms in Computer Vision

Johannes Dienst — Tue, 09 Jul 2024 10:14:42 +0000

When you read about Artificial Intelligence, Machine Learning and Computer Vision you often come across terms that seem to be common.

But do you know all of them and what they actually mean?

In this blog we will list the most common terms that we use in our day-to-day work.

Bounding Box

A Bounding Box is a rectangle around a detected element or region in a picture. In code they are represented with their coordinates. There are different formats used. Some examples:

pascal_voc: [xmin, ymin, ymax, ymax]
coco: [xmin, ymin, width, height]
yolo: normalized [xcenter, ycenter, width, height]

Example in JSON:

{
  "xmin": "1128.2720982142857",
  "ymin": "160.21332310267857",
  "xmax": "1178.8204241071428",
  "ymax": "180.83512834821428"
}

Annotation

An annotation can be the visual representation of a bounding box or just a keypoint. But also a textual description is possible.

Usually bounding boxes are used in grounding together with labels and to establish a ground-truth or reference for trainingdata and testdata for training computer vision models.

Annotations differ from labels in that they can include any additional metadata.

Label

A label is given to a bounding box or a keypoint and identifies which class or category an object or area the model assigned to it. In the above example the person-text above the bounding box is a label.

Labels are also used to train models where the training set consists of correctly labeled images.

Image Classification

Image classification is used to put a predefined label on an image.

For an image of a cat would get the label cat, while the image of a sunset would get the label sunset.

This seems like a trivial thing to do but it actually has a lot of practical use cases:

Recognition of objects or entities depicted in images
Classifying medical image for detecting abnormalities
Image retrieval based on categories

Image Segmentation

The goal of image segmentation is to divide an image into areas that belong together. The areas are visually distinctive from each other.

The output of the process is a set of segmentation masks that can be used to process the image further.

Take this segmentation of an avatar image where the masks define where a person is and where the background is. A practical application would be to remove the background based on these masks.

Object Detection

Object detection is used to identify objects in images and video. It is a crucial task in a lot of domains such as Surveillance and Security, Environmental Monitoring and Augmented Reality.

It takes visual input and determines the objects that are present. The objects are described with a bounding box and also get a class-label and sometimes a confidence score.

Feature

A distinctive feature in an area or keypoint of an image. Usually you attribute features to a specific class you want to detect.

For a mouse those could be the shape, surroundings or fur pattern. Those are represented as numerical vectors, called descriptors, that capture the visual properties of an area or keypoint.

Feature Vector

Feature vectors are not limited to Computer Vision, but used there for all kind of tasks.

A feature vector is generated from input data such as an image and contains one numerical entry for every feature. Feature vectors are typically high-dimensional.

In practice feature vectors get generated by a Backbone (see next heading) which are by themselves models trained to extract useful features.

Grounding

Grounding is the process of connecting high-level concepts, for example Mouse, with visual features, for example tail.

Now that you have high-level concepts and their representation in the model you can also put them into context and relation with each-other.

Grounding can also be done by humans for training data. There you annotate an input with bounding boxes and labels, so the model has a Ground Truth to learn from.

Referring Expression

Referring Expression in Computer Vision are used to describe objects or areas in images. They can contain attributes like color or relations to other objects.

Take this example: "The bike next to the child." You do not point to the bike directly but give a reference point child. Or you could give a description: "The red building next to the blue car.".

Referring Expressions are typically used in tasks like object localization, object detection or image retrieval.

Zero-Shot

In general zero-shot is when models can do tasks they were not specifically trained on.

To some extend, and most of them rather poorly, every model can perform tasks without specific training as the goal of training is to get a higher-level understanding of concepts.

Zero-shot learning in Computer Vision also includes more information than just labeled examples. The focus is on data that describes relationships between categories, so the model can generalize better to new categories.

This also requires large scale training on a lot of data, so the model is exposed to billions of examples and can generalize without specific training to new tasks.

Conclusion

That are all, but sure there are more. Do you know of any we should include?

Let us know on Social Media.

Tutorial: Automate Highlighting in WYSIWIG Editors

Johannes Dienst — Tue, 26 Sep 2023 22:00:00 +0000

Automate Highlighting in Wysiwyg Editors

Working in What-You-See-Is-What-You-Get editors (WYSIWYG) can be very tiring and also taxing on your wrists. Things like highlighting the same thing over and over again can often not be done with search-and-replace, because there is just no button for it 😤

I recently had to put a transcript of a podcast into such an editor and had to bold the speaker hints, so that you clearly know who was speaking in that part of the podcast. It looks like this:

JD: Welcome to J^2 Talk AI

JH: I think first of all we should understand...

It is always alternating between JD and JH as that is the nature of a conversation 😋.

Bolden the beginning speaker hints would need me to double-left-click the abbreviations and then hit the shortcut cmd+b (I am on macOS) in this particular editor.

I do not want to do this by myself as that would mean to do roughly 20*2 = 40 actions on average per episode. And I had 5 episodes... Extremely straining on your wrists and arms and it is dull work also!

In this tutorial I will show you how I automated this task with AskUI and also how to work around some kinks of WYSIWIG-Editors like microanimations and strange behaviour.

Prerequisites

AskUI installed: Windows, Linux, macOS
Open your favorite WYSIWIG-Editor

The Problem

So what do we need to do?

Find all the JD and JH
Double-left-click each one
Press cmd+b (macOS) or (ctrl+b Linux and Windows)

This is how it looks:

Let Us Develop It Step-By-Step

So the first thing that needs to be done is finding all the JD and JH. This can be done with the get() command. As you can see in the demo above there are possibly multiple pages, so we have to scroll somehow. The easiest way to do this is to press the pagedown-key. So let us be conservative and press that key three times.

Every time we press it there is a scroll animation that might mess up the inference. So let's wait two seconds with waitFor(2000). That should be enough to finish the animation.

for (let i = 0; i < 3; i++) {
  const jds = await aui.get().text().containsText("JD:").exec();

  const jhs = await aui.get().text().containsText("JH:").exec();

  await aui.pressKey("pagedown").exec();
  await aui.waitFor(2000).exec();
}

This already looks like it would be a good idea to refactor it and put the actual highlighting code into a function highlight().

for (let i = 0; i < 3; i++) {
  highlight();

  await aui.pressKey("pagedown").exec();
  await aui.waitFor(2000).exec();
}

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();

    const jhs = await aui.get().text().containsText("JH:").exec();
}

Great, now we need to mouse-double-left-click the text JD and JH one by one with a for-loop. But when we try it with await aui.moveMouseTo(jds[0]).exec() it will move the cursor to the middle of the line where JD/JH is at the beginning. We have to use the bounding box x-min and y-min values to get to the beginning of the line. Also we have to add like 10 pixels to the values to actually move the cursor over the JD/JH.

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();
    for (let i = 0; i < jds.length; i++) {
        const bbox = jds[i].bndbox;
        await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();
    }

    ...
}

Now we can do the highlighting by mouse-double-left-click and press the bold shortcut.

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();
    for (let i = 0; i < jds.length; i++) {
        const bbox = jds[i].bndbox;
        await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();
        await aui.mouseDoubleLeftClick().exec();
        await aui.pressTwoKeys("command", "b").exec(); // Use CTRL+b on Linux/Windows
    }

    ...
}

That should highlight all the JD. Let's do the same with JH. Also we need to add a keypress of escape here because the mini-menu that pops up when double-left-clicking the last JD might cover a JH

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();
    for (let i = 0; i < jds.length; i++) {
        const bbox = jds[i].bndbox;
        await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();
        await aui.mouseDoubleLeftClick().exec();
        await aui.pressTwoKeys("command", "b").exec();
    }

    await aui.pressKey("escape").exec();

    const jhs = await aui.get().text().containsText("JH:").exec();
    for (let i = 0; i < jhs.length; i++) {
        const bbox = jhs[i].bndbox;
        await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();
        await aui.mouseDoubleLeftClick().exec();
        await aui.pressTwoKeys("command", "b").exec();
    }
}

Two More Tweaks - Complete Code

Another problem you might run into is missing focus on the editor. This is why adding the first two lines to your code is beneficial. It moves the mouse cursor into the editor and with the left-click gets the focus.

Also extracting a function boldenText(bbox) that does the highlighting given a bounding box bbox avoids duplicate code.

Here it is, the complete code:

it('highlights all the JD and JH', async () => {
    await aui.moveMouse(500, 500).exec();
    await aui.mouseLeftClick().exec();

    for (let i = 0; i < 3; i++) {
        await highlight();
        await aui.pressKey("pagedown").exec();
        await aui.waitFor(2000).exec();
    }

})

async function highlight() {
    const jds = await aui.get().text().containsText("JD:").exec();
    for (let i = 0; i < jds.length; i++) {
        await boldenText(jds[i].bndbox);
    }

    await aui.pressKey("escape").exec();

    const jhs = await aui.get().text().containsText("JH:").exec();
    for (let i = 0; i < jhs.length; i++) {
        await boldenText(jhs[i].bndbox);
    }
}

async function boldenText(bbox) {
  await aui.moveMouse(bbox.xmin + 10, bbox.ymin + 10).exec();

  await aui.mouseLeftClick().exec();
  await aui.mouseDoubleLeftClick().exec();
  await aui.pressTwoKeys("command", "b").exec();
}

Conclusion

In this tutorial you learned how to automate tedious tasks like highlighting some recurring text in a WYSIWYG-Editor.

You also learned how to overcome some of the challenges user interfaces throw at you, like mini-popups and micro-animations.

Photo by Pankaj Patel on Unsplash

A Comprehensive DevRel Metrics Guide (Part 4): Company Stages and Reporting

Johannes Dienst — Mon, 25 Sep 2023 06:00:00 +0000

Imagine you are working at an early stage startup and you are tasked with tracking meaningful metrics. And then imagine you are working at an enterprise company with the same task.

Would you track the same metrics?

I have been scratching my head over this question because it does not make sense to me that the metrics should be the same for both companies 🤔.

What Company Stages Are There?

I found an inspiring talk from Vera Tiago that resonated with me. There is also a written form of it, if that is more your thing 🥰.

In there she talks about North Star metrics and then comes to the topic of DevRel-Focus areas. Her Developer Relations focus areas are:

Awareness
Adoption
Education
Retention

And then she makes an interesting distinction between two areas you have to look at as a company:

Maturity of your product
Maturity of your community

So if you are an early stage startup you probably have an unmature product and a just forming community. Thus working towards retention may not be your highest priority as your product has to many kinks and your community is too small.

It should be better to work towards acquisition and adoption to get more users.

By doing this you get more feedback and exposure to achieve product market fit.

Vera Tiagos four pillars of DevRel and company stage (Source Vera Tiagos Talk above in written form).

Evaluating and validating your company’s stage and your focus with management is extremely important as it again will determine the metrics you are prioritizing.

How to Do Reporting?

Many companies have a monthly All-Hands meeting. Some call it a Jour-Fix. Whatever you call it, entering your collected data into a nice presentation template gives you the possibility to always be ready to give a presentation.

I do not get into all the details in the presentation deck but focus on the things that matter now. Even when I track more than that as outlined in the previous blog post. There is a dedicated slide at the end where I link to the raw metrics if anyone is interested in them.

With the slide deck I am good to go to present 5-10 Minutes and give a concise update about how we are doing 🎉

Here is my template as a Google-Slide Deck.

Conclusion

And this marks the end of this series. We discussed why metrics are important, where to even start and my personal view on what metrics to track.

In this blog post I gave an introduction on what might be important and expected from DevRel at different company stages.

As always: Do not blindly follow advice but take it as a food for thought and an idea to test in your circumstance! I feel this is especially important in DevRel as no company is the same 🙃.

Photo by Andre Taissin on Unsplash

Visual Regression Testing with AskUI and Jest

Johannes Dienst — Mon, 07 Aug 2023 07:07:00 +0000

Visual Regression Testing (VRT) is a useful method to guard yourself against unwanted layout and visual changes that can occur with new version of a Graphical User Interface (GUI).

AskUI is not a testing tool in the original sense but an automation tool for all GUIs that is platform-independent. Wouldn't it be nice to combine VRT with AskUI, so you do not have to maintain two different projects.

Luckily you can do this because Jest - AskUIs favourite runner - can be extended to provide VRT, while AskUI delivers the screenshots to test on.

In this blog you will learn how to use AskUI in combination with Jest and the package jest-image-snapshot to include a visual regression test in your AskUI runs.

Prerequisites

AskUI installed: Windows, Linux, macOS

Setup `jest-image-snapshot`

First you need to install the package that provides the VRT capabilities: jest-image-snapshot.

Installation

Install it as a dev dependency:

npm install --save-dev jest-image-snapshot

Integration

The package provides a function toMatchImageSnapshot which implements Jest's Matchers<R> interface making it a Jest matcher that can be used with Jest's expect().

We have to add this matcher to Jest with expect.extend like this in your workflow file (See the docs):

const { toMatchImageSnapshot } = require('jest-image-snapshot');
expect.extend({ toMatchImageSnapshot });

Now the only thing you have to provide for a regression test in your workflows is an image to the function expect() as Buffer:

expect(imageBuffer).toMatchImageSnapshot();

You can use AskUI to get the screenshot as a base64 encoded string. The string is in the format of a data URL. So you have to strip the scheme data:[<mediatype>][;base64], away because it is not a valid image with the scheme. Here is the full code:

const annotate = await aui.annotate();
const imageBase64 = annotate.image;

// Strip away this: data:image/png;base64,
// Not valid png -> Only if used as a data URL
let imageBuffer = Buffer.from(
    imageBase64.split('base64,')[1], 'base64');

expect(imageBuffer).toMatchImageSnapshot();

How to Get it to work with TypeScript

When you run this you will get an error:

FAIL  test/my-first-askui-test-suite.test.ts
  ● Test suite failed to run

  test/my-first-askui-test-suite.test.ts:16:25 - error TS2551: Property 'toMatchImageSnapshot' does not exist on type 'JestMatchers<Buffer>'. Did you mean 'toMatchSnapshot'?

  16     expect(imageBuffer).toMatchImageSnapshot();
                            ~~~~~~~~~~~~~~~~~~~~

    node_modules/@types/jest/index.d.ts:1111:9
    1111         toMatchSnapshot<U extends { [P in keyof T]: any }>(propertyMatchers: Partial<U>, snapshotName?: string): R;
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      'toMatchSnapshot' is declared here.

This is because Jest's typings in jest.d.ts do not include toMatchImageSnapshot.

Luckily TypeScript has a mechanism called Declaration Merging. When we create a file jest.d.ts in our project and declare our matcher there, TypeScript will pick it up.

declare namespace jest {
    interface Matchers<R> {
      toMatchImageSnapshot(): CustomMatcherResult;
    }
}

When you rerun it you will get no error 🥳.

How `jest-image-snapshot` Works

From the repository:

Given an image (Buffer instance with PNG image data) the toMatchImageSnapshot() matcher will create a image_snapshots directory in the directory the test is in and will store the baseline snapshot image there on the first run. Note that if the customSnapshotsDir option is given then it will store baseline snapshot there instead.

On subsequent test runs the matcher will compare the image being passed against the stored snapshot.

To update the stored image snapshot run Jest with --updateSnapshot or -u argument. All this works the same way as Jest snapshots.

You also want to check out the options you can provide to toMatchImageSnapshot() to fine-tune the behaviour. For example you might want to set a threshold for a mismatch so that minimal differences do not fail a run:

expect(image).toMatchImageSnapshot({
    failureThreshold: 0.01,
    failureThresholdType: 'percent'
  });

Conclusion

You can use AskUI in combination with a Visual Regression Testing library that integrates with Jest like jest-image-snapshot.

With this you can validate your GUI's final state and ensure it is the one you expected 🥳.

Integrating Test Data in AskUI Workflows from APIs, CSV Files, and Databases

Johannes Dienst — Tue, 11 Jul 2023 09:39:43 +0000

Are you struggling to integrate test data into your AskUI workflows effectively? Do you find yourself juggling APIs, CSV files, and databases without a clear approach?

Integrating test data from various sources can be a daunting task.

By the end of this post, you'll have a foundation for integrating test data into your AskUI workflows using APIs, CSV files, and databases.

Integrate API

To use an API for fetching test data, you'll need a library like axios. After installing it, import it into your AskUI workflow file and use it to call an API, extract data, and insert it into a text field.

// Top of your workflow file
import axios from 'axios';

const url = 'https://fruityvice.com/api/fruit/all';
const data = await axios.get(url);
await aui.typeIn(data.data[0].name)
         .textfield()
         .contains()
         .text()
         .withText('E-Mail Address')
         .exec();

Reading from CSV Files

For integrating CSV files and databases, let's dive into a combined example. Begin by installing the required libraries and adding the necessary imports to your AskUIworkflow file. First, read the CSV file into your code and parse it:

// Top of your workflow file
import * as fs from 'fs';
import csv from 'async-csv';

const csvData = fs.readFileSync("./data.csv");
const rows = await csv.parse(csvData);

Our CSV data looks like this. Save it as data.csv in the root-folder of your AskUI project:

year_month,month_of_release,passenger_type,direction,sex,age,estimate,standard_error,status
2001-01,2020-09,Long-term migrant,Arrivals,Female,0-4 years,344,0,Final
2001-01,2020-09,Long-term migrant,Arrivals,Male,0-4 years,341,0,Final
2001-01,2020-09,Long-term migrant,Arrivals,Female,10-14 years,459,0,Final
2001-01,2020-09,Long-term migrant,Arrivals,Male,10-14 years,510,0,Final
2001-01,2020-09,Long-term migrant,Arrivals,Female,15-19 years,899,0,Final
2001-01,2020-09,Long-term migrant,Arrivals,Male,15-19 years,904,0,Final

Create and Initialize Database

Next, create a database on your file system, within the same project as your Ascii workflow file. Set up a schema by opening the database and creating a table:

// Top of your workflow file
import * as aadb from 'aa-sqlite';

// Open a new database
const filepath = "./data.db";
await aadb.open(filepath);

// Create a table
await aadb.run(`
  CREATE TABLE migration (
    year_month VARCHAR(10),
    month_of_release VARCHAR(10),
    passenger_type VARCHAR(50),
    direction VARCHAR(20),
    sex VARCHAR(10),
    age VARCHAR(50),
    estimate INT
  )
`);

Once completed, you'll have successfully integrated test data into your AskUI workflow using both CSV files and databases.

Insert Data into Database

Now that we have our database and table set up, we'll insert the parsed CSV data into the database. To do this, iterate over each row of the CSV data and use a SQL statement to insert it:

for (const row of rows) {
  await aadb.run(`
    INSERT INTO migration (
      year_month, month_of_release, passenger_type,
      direction, sex, age, estimate
    ) VALUES (?, ?, ?, ?, ?, ?, ?)`,
    row.year_month, row.month_of_release, row.passenger_type,
    row.direction, row.sex, row.age, row.estimate
  );
}

If you don't have a CSV file, you can follow a similar approach to the previous example, where you read data directly from the database and integrate it into your code like this:

// Get all the data
const dataAsync = await aadb.all('select * from migration;')
console.log(dataAsync);

Conclusion

We provided examples of incorporating test data into your AskUI workflow using APIs, CSV files, or databases. While these examples are not comprehensive, they serve as a solid starting point for integrating test data into your projects.

Being a Developer Advocate: Half a Year Review

Johannes Dienst — Fri, 30 Jun 2023 14:14:28 +0000

I blinked and six months are already gone since I plunged head first into Developer Relations as a single Developer Advocate at a small startup.

In hindsight it was clear that it would have been a lot easier with a whole team already in place to learn from 😋

But at that time it was exactly what I needed: The freedom to learn in every direction possible and expand my knowledge in all of the ways in an area I deeply care about!

I got to do so many things:

Building Developer Docs from the ground up using Docs-As-Code
Giving a ton of presentations at conferences and on company workshops
Hosting events
Sketchnoting our Meetup
Hosting a podcast and learning about the distribution of it
Creating a 15 Episodes strong YouTube short-form series about the companies product by doing everything from scripting to thumbnails to postproduction and promotion
Writing a ton of technical blog posts
Passionately advocating internally for our users
Improving the developer experience as much as I possibly could

What I most enjoyed?

Helping humans to succeed is the best possible feeling you can get in my opinion 🥰

Doing it while also being allowed to work with code and docs is the perfect combination! Nothing beats being able to geek out about machine-learning stuff on a podcast, then getting feedback from users on how to improve your product afterwards. Finding a quick fix for it you can document quickly in the docs and then communicate the overall problem to product/engineering for improvement.

It is this mix of cross-functional skills and impact you can have when you do it well that feels like a superpower.

At the same time it can make you feel helpless because all you can do is try to influence without the assurance it will actually achieve your desired outcome.

What I hope to achieve in the next 6 months?

So now that I know what I enjoy and hopefully also excel at, I will work hard to professionalize it.

At the moment a lot of things are not automated and time consuming. Publishing blog posts for example means publishing them manually on every platform. Then doing the social media posts manually.

I also hope to build more. By learning all the basics building time with our tool and also coding in general was not enough. Not losing touch with coding is a major challenge I want to solve by actually building stuff consistently.

Also a goal will be to do a talk at a meetup or if even possible at a conference about DevRel. I really enjoy geeking out about docs and metrics and I believe this is something to explore further and talk about.

So if you want to geek out about DevRel, docs, metrics or anything else: Hit me up and I am happy to join!

Use the Power Of TypeScript in AskUI

Johannes Dienst — Fri, 30 Jun 2023 13:28:28 +0000

Optimizing UI Automation with Functions in TypeScript

In the past, we've discussed using if and else statements, try and catch blocks, and for loops in TypeScript for building UI automations.

However, to avoid repeating code and add real UI interactivity to your AskUI workflows, it's essential to harness the power of functions. Functions can help create reusable and efficient code for checking if elements are present and reacting accordingly.

The Problem

When you want to check if an element is present on your UI you might do it like this and do something else when the element is not found:

try {
    await aui.expect().text('<Your text here!>').exists().exec();
  } catch(error) {
    // Do something else
  }

This is totally fine, but once you create more sophisticated workflows you will notice that you need a variation of this code 1,2,3 ... x times.

You can do better if you extract this into a function!

Extract into a function

To get started, create a util.ts file in the same folder as your automation file. First, import the UiControlClient into this file:

import { aui } from './helper/jest.setup';

Then you can create a function in this file that takes a parameter text and returns true or false depending on if the element is present or not. This function is asynchronous and checks for the presence of a specific element on the screen. It returns a boolean value to indicate whether the element is present or not.

export async function checkSuccess(text: string) {
  try {
    await aui.expect().text().withText(text).exists().exec();
    return true;
  } catch(error) {
    console.log(`${text} not found!`);
  }
  return false;
}

The export keyword makes sure we can use it in our actual AskUI-workflow file.

// Do this in your actual AskUI workflow file at the start
import {checkSuccess} from './utils';

// Usage
if ((await checkSuccess('Success')) === false) {
  // React to text not being there
};

Implementing it like this makes it possible to reuse the this functionality for all text you need. It also improves the readability of your workflow file 🥳

Endless Possibilities

You can wrap (nearly) any functionality you need in a handy function.

Here's an example of a waitUntil() function that you can use to wait for an element or an app to appear on the screen, even if it takes longer than a few seconds:

// Retry the command 5 times with a
// wait time of 2 seconds between each try
async function waitUntil(askuiCommand: Promise<void>, maxTry = 5) {
    try {
        await askuiCommand;
    }
    catch (error) {
        if (maxTry == 0) {
            throw error;
        }
        console.log(`Retry predicting command, ${maxTry} tries left`);
        await aui.waitFor(2000).exec();
        await waitUntil(askuiCommand, maxTry - 1);
    }
}

// Wait for the text 'Github' to be displayed
await waitUntil(
    aui.expect().text().withText('Github').exists().exec();
)

Conclusion

Leveraging functions in TypeScript significantly streamlines your UI automation efforts and provides numerous opportunities for optimization.

A Comprehensive DevRel Metrics Guide (Part 3): A Framework to Get Started!

Johannes Dienst — Fri, 30 Jun 2023 13:05:56 +0000

In part two of the series we discussed what DevRel north star metrics commonly get used across the industry. We also saw that it can be quite tricky to find the right metrics to measure because they are plenty, sometimes lag behind or the relation to your North Star is not direct but only delivered over time.

So what should we do?

I will answer this question in this article for people who are just starting out and looking for a way to have a solid foundation to build on. Because that is what I found myself in when I started my Developer Advocate journey 😊.

But also if you have something in place already. This article will help you make sense of metrics and give you inspiration.

The Solid Framework: Kim Maidas Keystone Framework

Just starting out with DevRel-Metrics? Look no further than Kim Maidas DevRel Metrics Keystone Framework. Over the course of her career leading in Developer Relations she says she found a way to holistically measure DevRel in a way the C-Suite understands. And I could not agree more!

Kim Maidas Keystone DevRel Metrics

Her idea is to define one Keystone (North Star) as your guidance. Her recommended keystone is 1 signup/registration. Ideally it is your companies north star.

Once you have your keystone you can come up with a price tag for it. Yes, real money!

Then with a price tag you can attach a percentage of this keystone value to each metric in the following categories:

There is an order to it also. Reach metrics are at the top of the funnel and thus improving metrics from that category is not as valuable as further down the funnel. Engagement and finally DevRel Qualified Leads are worth so much more and you can add more value to each metric in those categories.

What you basically do is attaching a value tag on every metric and are thus able to calculate the value DevRel brings to the table 🥳

I do not want to copy Kim Maidas article, so for the sake of brevity, I encourage you to read it and use her method to give every metric a meaning tied to company metrics.

There is also much more in her article that makes the read worthwile!

My Personal Additions

I personally found two additions helpful in showing impact on product/engineering:

Internal Impact
Direct Impact.

The first one captures the feedback you bring from the community into the company. The second one measures your direct impact you have by doing developer experience audits and creating content.

Internal Impact

In my daily work a lot of things I do are related to product development. Be it providing feedback for new features, betatesting new features, writing docs, gathering feedback from the community and relaying it back to development. The above areas do not cover them completely.

Luckily I found two articles that cover these areas well (See DevRel Metrics and Why they Matter by Sean Falconer and The Core competencies of Developer Relations by Reto Meier). They talk about the developer relations cycle where they introduce Developer Feedback and Developer Sentiment that Developer Relations collects and brings into product development.

The Developer Relations ongoing interface cycle (Source The Core competencies of Developer Relations by Reto Meier)

As you are acting as a middlehuman for these two, I call this internal impact as you affect your product through the feedback delivered.

What I like to measure here is the number of feedback and sentiment brought to the department in your company where it is best seated. If anything comes out of this feedback that is also noted down.

Direct Impact

As you have noticed, most metrics are loosely tied to company goals. But there are ways to demonstrate a direct impact. Things I found useful are the following:

Developer Experience Audits: Evaluating your product as a user would and providing detailed feedback and improvement suggestions to product management. You should also track what is finally implemented and if possible if any metric was impacted by the change (Less bug-tickets, specific user complaints decreasing).
Tracking your content's impact: With sufficient tracking in place you can track if your company’s North Star metric is affected by your content like docs, blogs or videos. In fact this is what I do and I could demonstrate that videos on YouTube drive Sign Ups.

Conclusion Part 3

With a framework like the one from Kim Maida you can lay a solid foundation for all your metric needs in a structured way. It also makes reporting easier by directly tying individual metrics to your companies north star.

In the last part of the series we will dive into the wonderful world of company development stages and their individual needs from DevRel. Early stage startups are different than enterprise organizations. And there is a lot of space in between.

Thus we should shift our focus and the metrics we take into focus accordingly.

I will also share my template I use to report DevRel metrics every month.

Photo by Leonardo Vazquez

A Comprehensive DevRel Metrics Guide (Part 2): DevRel Metrics to Track?

Johannes Dienst — Thu, 15 Jun 2023 12:47:34 +0000

Now that we got the Why? out of the way. Let's face the difficult challenge on what DevRel-Metrics we should collect in the first place.

Of course I will stand on the shoulders of giants here again as a lot of smart DevRel-practitioners have already written about this.

The goal of this post is to get to a common understanding what others usually track. I also want to give you my opinion on metrics as I think there is so much confusion about metrics. I believe that a pragmatic approach is reasonable to be practical for each individual situation.

We will discuss the following things:

DevRel-North Star metrics: Why they are useful but not sufficient
Common metrics that are collected in the industry
My critique and opinion about this

DevRel-North Star Metrics

Swyx claims nearly all companies settle on one specific metric: Monthly Active Developers (Source: Measuring DevRel by swyx)

Also popular:

Registered Developer Accounts (Source: Slashdata Report 2020)
Developer Satisfaction score (Source: Slashdata Report 2020)
Time To Hello World (TTHW) (Source: Sean Falconers - DevRel Metrics And Why They Matter)
Time to First WOW! Exclamation (TTFWE) (My own invention: The first time a user experiences and realizes the value of your product.)
Number of signups / registrations (Source: Kim Maidas Keystone Framework)
...

Also, have a look at DevRel-KPIs.com for more inspiration.

Common Metrics in the Industry

So this may be useful when you have to report one or two numbers to management, but not sufficient for what DevRel is actually doing in their day-to-day work.

Many more metrics have been collected and I will just link them here for you to explore and get an idea:

I really like the DevRel-Metrics swyx compiled for his North Start-metric Monthly Active Developers

Metrics that feed into Monthly Active Developers North Star Metric.(Source: [Measuring DevRel by swyx](https://www.swyx.io/measuring-devrel))

From Kim Maidas Keystone Framework:

Also again DevRel-KPIs.com is a great resource to get you going.

Are There Bad Metrics

I would say It depends :-P

Usually you do not want to be measured with a marketing metric. We are dealing with relations here, remember?
But if we produce content for example that is for creating awareness or inspiration. What metrics can we actually collect to measure its success? In my opinion this would be something like views or likes or Github-stars.

So I suggest the following: Be pragmatic with what you track. There is no harm in collecting a marketing metric if it fits your current situation. If management cares about these sometimes so called Vanity Metrics, even better. Easy to collect and report on.

But do not get caught up in them! An audience is not a community and you want to foster real relationships :-)

My Critique and Opinion About This

It always felt wrong as a software developer to reduce a complex topic into a single number. It feels even more so with DevRel. I get that management would like to have a single number to gauge how DevRel is doing. But DevRel performs a cross-functional function in a company. The metrics are influenced by a lot of departments and also ultimately by the state your product is in.

A metric like Monthly Active Developers is a great metric if the management trusts that you influence it positively. The problem is, that it is usually influenced by a lot of departments like product, support, technical writing, marketing and so forth. If your product has sharp edges you can not work around even with the best onboarding and with the best feedback for engineering you can possibly provide: Good luck with improving your TTHW!

Also tieing your activities to a North Star metric may sometimes be hard as it can lag significantly behind as not every relationship or piece of content has impact immediately. Often it is a compounding effect of positive encounters that tips the scale.

Conclusion Part 2

In this post we discussed popular DevRel-North Star metrics like Monthly Active Developers used in the industry. They are useful when reporting to management, but not sufficient to measure the complex day-to-day work or specific project/campaigns DevRel is focusing on in a specific company.

We gave a lot of pointers on what you can also collect with the general recommmendation to be pragmatic and, if possible, to focus on metrics that cover relationships.

In the next part of this series we will cover Kim Maidas Keystone Framework. It is a great framework if you are just starting out! I use it with a few extensions. It is useful in my current situation and it includes the Relations part of a DevRel-Practitioner!

Photo by Igor Mashkov: https://www.pexels.com/photo/radio-telescope-against-sky-with-stars-6325001/

DEV Community: Johannes Dienst

Use Playwright and AskUI Together

Use Playwright and AskUI Together

Prerequisites

Install Playwright

Configure Playwright

Write a Workflow

Testrun

Conclusion

Run AskUI Workflows with Pipedream for Smoke Testing

Prerequisites

What Will We Build? Aka the Use Case

Our Use Case: Smoke Test with Visual Selectors

1. Add a Trigger

2. Prepare a Custom Code Action

3. Write the AskUI Workflow

4. Send an Email

Gitpod As Remote Machine

Conclusion

Use AskUI and Cucumber Together

Use AskUI and Cucumber Together

Prerequisites

Prepare Setup

1. Change Jest's testEnvironmentOptions

2. Tell Jest Where to Find the Implementation for The Step Definitions

Install jest-cucumber

Create a Basic Features File

Create the Step Definitions Implementations

Run The Workflow

Conclusion

Common Terms in Computer Vision

Bounding Box

Annotation

Label

Image Classification

Image Segmentation

Object Detection

Feature

Feature Vector

Grounding

Referring Expression

Zero-Shot

Conclusion

Tutorial: Automate Highlighting in WYSIWIG Editors

Automate Highlighting in Wysiwyg Editors

Prerequisites

The Problem

Let Us Develop It Step-By-Step

Two More Tweaks - Complete Code

Conclusion

A Comprehensive DevRel Metrics Guide (Part 4): Company Stages and Reporting

What Company Stages Are There?

How to Do Reporting?

Conclusion

Visual Regression Testing with AskUI and Jest

Prerequisites

Setup jest-image-snapshot

Installation

Integration

How to Get it to work with TypeScript

How jest-image-snapshot Works

Conclusion

Integrating Test Data in AskUI Workflows from APIs, CSV Files, and Databases

Integrate API

Reading from CSV Files

Create and Initialize Database

Insert Data into Database

Conclusion

Being a Developer Advocate: Half a Year Review

What I most enjoyed?

What I hope to achieve in the next 6 months?

Use the Power Of TypeScript in AskUI

Optimizing UI Automation with Functions in TypeScript

The Problem

Extract into a function

Endless Possibilities

Conclusion

A Comprehensive DevRel Metrics Guide (Part 3): A Framework to Get Started!

The Solid Framework: Kim Maidas Keystone Framework

My Personal Additions

1. Change Jest's `testEnvironmentOptions`

Install `jest-cucumber`

Setup `jest-image-snapshot`

How `jest-image-snapshot` Works