DEV Community: kayYOLO

Clicknium Cookbook

kayYOLO — Thu, 09 Mar 2023 12:03:45 +0000

Clicknium is a Python UI automation library used primarily for automating Windows desktop applications and web applications. As the documents are hard to learn the tool quickly, this article will provide a systematic guide on how to use Clicknium.

1. Basic Concepts

Clicknium generates a locator by recording mouse clicks on UI elements, which stores various properties of the UI element. This allows Clicknium to locate the corresponding UI element by using the Locator. Once the UI element is located, Clicknium provides various common methods for operating UI elements, such as inputting (set_text), mouse clicks (Click), etc. With these methods, Python developers can write UI automation scripts easily.

Clicknium Development Kit

The Clicknium Python SDK is the automation core of Clicknium. Around this core, Clicknium provides a series of plug-ins to assist in developing automation scripts:

Clicknium VS Code plug-in: manage and debug locators, provide Python code completion and intelligent prompts.
Clicknium browser plug-in: enhance web automation capabilities, support Chrome, Firefox, Edge, Brave, and Vivaldi.
Clicknium Recorder: supports recording UI elements to generate locators.

Clicknium Python SDK can be installed via pip install clicknium. After searching for Clicknium in the VS Code extension and installing the extension, various plug-ins for Clicknium and the SDK can be managed and updated in the extension.

2. Development Environment Configuration

Windows 7 SP1+ is required:
Search for Clicknium in the VS Code extension and install it:

Python module can also be installed by pip install clicknium.

3.Automation Script Development Process

3.1 Sample Script

First, take a look at the sample script. Open VS Code, enter Clicknium: Sample with Ctrl+Shift+P, and then select a new empty directory to store the sample project.

Now we have the following sample project. Press F5 to run and Ctrl+F5 to debug.

The directory structure of the sample project:

/.locator: Locator information

/.locator/sample_img: snapshot of the locator

/.locator/sample.cnstore: stores the properties of this Locator

Code analysis:

Part 1: Installing the browser plug-in

    if cc.edge.extension.install_or_update():
      print("Please open edge browser to enable clicknium extension, then run sample again.")

It is easy to understand that the corresponding browser plug-in needs to be installed.

Part 2：Web automation, automatically open your browser, go to bing search, enter clicknium in the input box, and click the search button.

    tab = cc.edge.open("https://www.bing.com/")
    tab.find_element(locator.sample.bing.search_sb_form_q).set_text('clicknium')
    tab.find_element(locator.sample.bing.svg).click()
    sleep(3)
    tab.close()

Line 1: Using clicknium to call the Edge browser and open the Bing homepage, returning the corresponding browser tab.

Line 2: Calling find_element within the tab and passing in the Locator of the input box as a parameter, then using the set_text API to input the text information.

Line 3: Calling find_element within the tab and passing in the Locator of the search button as a parameter, then using the click function.

Line 4&5: Waiting for 3 seconds, then closing the corresponding tab.

Through this code snippet, one can roughly understand the logic of clicknium. Clicknium provides functions such as find_element that accept the locator parameter to obtain UI elements. It also provides some common methods for operating UI, such as set_text, click, etc., to simulate human operations.

Part 3：Windows desktop automation. Open Notepad and enter "clicknium".

    process = subprocess.Popen("notepad")
    ui(locator.sample.notepad.document_15).set_text("clicknium")

The API for Desktop Applications and Web browsers is the same.

the Clicknium core: Locator

You might have no idea about how to create a locator since the sample is without a process. It will be introduced in the next part.

Developing Your Own Automation Scripts from Scratch

Starting from scratch, use the Chrome browser to search for the Clicknium website and automatically register for an account.

Creating a Project

Create a folder named ClickniumSample to store the Python Project, open the folder with VS Code, and create a Python file named Sample.py.

Open the Clicknium official website:The best python automation module | Clicknium

In VS Code, find the Capture button next to LOCATOR and launch the Clicknium Recorder.

Hold down Ctrl and click on the Sign Up button:

Upon success, the Locator for the registration button is obtained.

Locators can be organized in a folder structure or renamed by oneself. The structure is similar to that of namespaces, so special characters such as spaces cannot appear in the locator name.

After generating the Locator, we can use the method find_element to locate the corresponding UI element. There will be intelligent prompts after inputting the Locator:

When the Mouse is placed on the Locator, a locator snapshot will appear to help identify the UI element corresponding to the Locator, and validation and re-recording can also be done.

After locating the UI element, a series of operations are supported. Since the object is a button, we choose to click:

Using the same method, record the five input boxes and one button on the registration page:

The button uses the Click method, and the input boxes use set_text. Use F5 to run and check the effect.

from clicknium import clicknium as cc, Locator
from time import sleep

def main():
    cc.chrome.open("https://www.clicknium.com/")
    cc.find_element(locator.clicknium.button_sign_up).click()
    sleep(5)
    firstName = "Kay"
    lastName = "Whitlock"
    email = "youremail@youremail.com"
    password = "youpassword"
    cc.find_element(locator.clicknium.auth.text_firstname).set_text(firstName)
    cc.find_element(locator.clicknium.auth.text_lastname).set_text(lastName)
    cc.find_element(locator.clicknium.auth.text_email).set_text(email)
    cc.find_element(locator.clicknium.auth.password_password).set_text(password)
    cc.find_element(locator.clicknium.auth.password_confirmpassword).set_text(password)
    cc.find_element(locator.clicknium.button_sign_up).click()

if __name__ == "__main__":
    main()

4. Mouse and Keyboard Input

Clicknium's documentation provides a detailed description of the mouse and keyboard operations supported. Here is a brief introduction.

Mouse Input:

The mouse operations supported by Clicknium can be found at Mouse. Taking a single click as an example, let's take a look at the parameters supported by the click function.

    def click(
        self,
        mouse_button: Literal["left", "middle", "right"] = MouseButton.Left,
        mouse_location: MouseLocation = MouseLocation(),
        by: Union[Literal["default", "mouse-emulation", "control-invocation"], MouseActionBy] = MouseActionBy.Default,
        modifier_key: Literal["nonekey", "alt", "ctrl", "shift","win"]  = ModifierKey.NoneKey,
        timeout: int = 30
    ) -> None:        
        """
            Single click the target element.

            Parameters:

                mouse_button: The available values are: 'left', 'right' and 'center', default is 'left'.  

                mouse_location: it is set to define the position where the element is to be clicked. The default position is the center of the element.  

                by: Defines the method to click the UI element.  
                    mouse-emulation: click the target UI element by simulating Mouse.  
                    control-invocation: click the target UI element by invoking its UI method. It may not be supported if it is a Windows desktop element.  
                    default: automatically choose method per element type. For Web element, use `control-invocation`; for Window element, use `mouse-emulation`.  

                modifier_key: The modifier key("alt", "ctrl", "shift", "win") to be pressed along with click, and the default is none.  

                timeout: timeout for the operation, the unit is second, and default is set to 30 seconds.  

            Returns:
                None
        """

mouse_button: The left or right button of the Mouse.

mouselocation: Absolute or relative position.

by: There are two modes: mouse-emulation and control-invocation. The names may sound abstract, but the former simulates the behavior of the Mouse, moving the Mouse and clicking it, while the latter uses a system call-like method invocation. In short, mouse-emulation has a higher success rate, while control-invocation has better performance. For desktop automation, it is recommended to use mouse-emulation first, and for browser automation, try control-invocation first and use mouse-emulation if it doesn't work.

modifier_key: Achieve the effect of holding down special buttons like alt, ctrl, shift while clicking the Mouse.

Keyboard Input

There are two commonly used keyboard inputs:

Inputting text
Pressing special buttons For inputting text, the set_text method is generally used:

    def set_text(
        self,
        text: str,        
        by: Union[Literal["default", "set-text", "sendkey-after-click", "sendkey-after-focus"], InputTextBy]= InputTextBy.Default,
        overwrite: bool = True,
        timeout: int = 30
    ) -> None:
        """
            Set text for the target element, it can be used to input text to a system.  

            Parameters:

                text[Requried]: text string to be input.                 

                by: the method to perform the text input operation. 
                    set-text: call system method to set text to the target element. Some Windows application elements may not be supported.  
                    sendkey-after-click: simulate Mouse click to activate the element, then send keys by simulating keyboard.  
                    sendkey-after-focus: set the target element to a focused state, then send keys by simulating keyboard.  
                    default: using different methods per target element type. `set-text` for web element and `sendkey-after-click` for desktop element.  

                overwrite: whether overwrite or append the text on the target element, default is True. 

                timeout: timeout for the operation. The unit of the parameter is seconds. The default is set to 30 seconds.

            Returns:
                No

Parameter Section:

text: The string to be entered.

by: set-text is a system call with the best performance and is generally used in browser scenarios.

sendkey-after-click: Simulates clicking on this UI element and then inputs by simulating the keyboard.

sendkey-after-focus: Sets the target UI element to focus and then inputs by simulating the keyboard.

overwrite: Can choose to append or overwrite directly.

Sendkey-after-click is a very commonly used option because, currently, input boxes often need to be clicked by the Mouse to become activated to receive input. They cannot receive input when in an inactive state. This is also the reason why input operations often fail in automation processes. It should be noted that this method is often hijacked by input methods, so when using this parameter, the input method should be disabled and switched to English input.

Pressing special buttons:
The send_hotkey method is mainly used.

    def send_hotkey(hotkey: str) -> None:
        """
            Send hotkey to the cursor's current position.

            Parameters:
                hotkey[Requried]: hotkey string represents one key or combined keys. For example, to represent the letter A, input string "A". To represent the letters A, B, and C, input paremeter "ABC". For special keys, please refer to [hotkeys](https://docs.microsoft.com/en-au/dotnet/api/system.windows.forms.sendkeys?view=windowsdesktop-6.0#remarks).

            Returns:
                None
        """

This method accepts a string as the hotkey code. The specific hotkey code corresponding to the keyboard button can be found in Microsoft's hotkey list. For example, if you need to press the Enter key, you can find the code for Enter as {ENTER} in the table.

cc.send_hotkey("{ENTER}")

5. Locator

The Locator is fundamental to locating UI elements. During the recording process, Clicknium Recorder reads various properties and structures of the UI elements, and then uses an algorithm to select the attributes that can help locate the UI element.

Returning to the directory structure in the Sample, a .locator folder will appear in the Python project directory, which stores the Locator data. VS Code reads the Locator data and displays it under the LOCATORS tab:

The above picture shows the Locator interface, where you can select the corresponding Locator under the LOCATORS tab to enter the Locator details page.

There are three buttons on the upper right side: Recapture, Validation, and Action.

Recapture: is mainly used to re-record the Locator after it becomes invalid. There is also an option called recapture&compare for comparison, especially when the Locator becomes invalid and needs to be re-recorded. This option can help identify which attributes have changed and caused the Locator to fail to locate the UI element.

Validation: is used to test whether the Locator can locate the corresponding UI element. Note that the webpage or app corresponding to the UI element needs to be open, and validation will not actively open it.

Action: is mainly used to test the usability of the Locator and perform some common UI operations, such as clicking (click), getting text (get_text), and setting text (set_text).

5.1 Locator Attributes

Locator attributes are actually the content of the Locator. When Clicknium Recorder captures an element, it automatically obtains the element's attributes and selects which attributes to use to locate the UI element. The content in the red box below the three shortcut buttons in the Locator interface is the Locator attributes. In the example in the above picture, there are three lines of attribute content from top to bottom. Clicknium finds the corresponding UI element through the hierarchical structure, and the three lines of attributes in the Locator actually represent the hierarchical relationship. The first line is the chrome.exe application. After locating the chrome application, the second line is the browser tab, and the third line locates the button through the web attribute.

5.2 Modifying Locator Attributes

In most cases, the attributes automatically selected by Clicknium can successfully locate the UI element. However, in some cases, we still need to fine-tune the Locator attributes to better match the UI element: first, choose the hierarchical relationship. If it is determined that a certain level of an attribute is not needed, you can uncheck the blue box in front of the attribute to exclude it from locating the element. When you click the hierarchical attribute with the Mouse, the attribute on the right will be expanded. For the attributes within the hierarchy, you can also choose whether to participate in locating. At the same time, in order to improve the generalization ability, the attribute value matching rule supports four modes: equals, startWith, endWith, and regex.

When the matching pattern is 'equals', wildcard characters are also supported. For example, name='test?_node', where '?' matches one character and * matches zero or more characters.

5.3 Parametric Locator:

For multiple UI elements of the same type, most locator attributes are the same, except for one or a few attribute values. In this case, creating a large number of locators is not feasible, especially when the number of elements is not fixed. This is where parametric locators are useful.

Using parameters in locators:

Use Mustache style by declaring a parameter with two curly braces and a variable name in the locator attribute:{{variable}}. The entire attribute value can be set to a variable."：

<Web ancestorId="{{id}}" tag="A" />

Or the part of the attribute value can be set as a variable：

from clicknium import clicknium as cc, Locator, ui
# replace variable'id' in parametric Locator during runtime
variables = {"id":"test"}
ui(locator.chrome.bing.search_sb_form_q, variables)

For example:

There is a list in the sample website：List group · Bootstrap v5.1

Use the Recorder to capture the fist item of the list, and get the Locator:

Modify the value of index by creating a variable: {{indexValue}} using double brackets, and then iterating through each item in the code with a loop.

from clicknium import clicknium as cc, Locator, ui

index = 1
driver = cc.chrome.open("https://getbootstrap.com/docs/5.1/components/list-group/")
while True:
    variables = {"indexValue":index}
    if driver.is_existing(locator.chrome.getbootstrap.li_anitem, variables):
        text = driver.find_element(locator.chrome.getbootstrap.li_anitem, variables).get_text()
        print(text)
        index += 1
    else:
        break

5.4 Get values of Locator attribute

The get_text method automatically retrieves the text displayed in the corresponding UI element. When we need to obtain other attribute values of the UI element, we can use the get_property method. Clicknium supports multiple automation technologies (refer to the Automation Concepts), and different automation technologies will generate different locators. Among them, Web corresponds to the browser, UIA and IA correspond to Windows desktop applications. For Web elements, on the Web automation page, you can find the list of properties (Web element properties).

from clicknium import clicknium as cc, Locator, ui
#get the URL of currentpage
tag = ui(locator.chrome.bing.search_sb_form_q).get_property("URL") 
# get the title of the UI element
tag = ui(locator.chrome.bing.search_sb_form_q).get_property("title")

6. Capture Similar elements

We often need to handle a large number of similar elements in web pages or applications, such as UI elements in lists and grids. Sequentially capturing these elements will generate a large number of locators, which is obviously not a good method. Some users even ask how to traverse a locator folder when the same type of Locator is placed in a folder. Of course, using parameterized locators mentioned earlier is also a way. However, on the one hand, you need to analyze which attribute and what value to traverse UI elements, and on the other hand, the range of parameters may be unknown, especially as the application becomes more and more complex and more and more dynamic loading is used, leading to the total number of UI elements to be captured cannot be predicted.

Usually, we use libraries like BeautifulSoup to parse the HTML structure of websites. As the website styles become more and more complex, the content captured is not always in the same structure. Each item may be wrapped in multiple layers of divs or other elements. Therefore, when parsing, we need to analyze the HTML structure and build an algorithm to find the smallest common node of all elements and traverse the structure. This process is indeed time-consuming but can be solved by algorithms.

Clicknium Recorder includes a feature for capturing similar elements. Users can capture two similar (same level) elements, and Clicknium's built-in algorithm automatically parses the HTML structure and calculates the minimum common node and hierarchical information to capture similar elements. If the calculated element is not what the user wants, a third capture can be performed, and the algorithm will recalculate the capture path based on the information of three UI elements.

6.1 how to use similar elements：

Using the Okay Bear image in Opensea as an example, which is a typical Grid structure.

a. Click the Capture button in the VS Code plug-in to start the Recorder:

b. Click on Similar elements in the Recorder:

c. Ctrl + Mouse click to grab the first and second elements.：

After crawling the Locator corresponding to the similar elements, call find_elements to return a UI element list (find_element returns a single UI element.

Sample code：

Get the picture URL by get_property. Download the picture using request lib, and use send_hotkey to press PgDn key to scroll and load more pictures.

import requests
from time import sleep
from clicknium import clicknium as cc, Locator
def main():
    cc.chrome.open("https://opensea.io/collection/okay-bears")
    nftDic = {}

    while True:
        newDic = {}
        finish = True
        sleep(4)
        nfts = cc.find_elements(locator.sample.opensea.img_okay_bear)
        for nft in nfts:
            src = nft.get_property("src")
            name = nft.get_property("alt")
            if name not in nftDic:
                newDic[name]=src
                finish = False

        for key,value in newDic.items():
            r = requests.get(value)
            print("Downloading:"+ key)
            imagePath = ".\image\" + key + ".jpg"
            with open(imagePath,'wb') as f:
                f.write(r.content)

        nftDic = nftDic | newDic
        if finish:
            break
        else:
            print("next page")
            cc.send_hotkey("{PGDN}")

if __name__ == "__main__":
    main()

7. Get structured data

In automated scenarios, there is a great demand for obtaining structured data, especially tabular data. Capturing similar elements is only suitable for capturing one column of data. Of course, one can capture a table column by column, but this approach may lead to alignment issues between different columns, increasing the risk of errors and making the operation very cumbersome.

There are two common scenarios for structured data:

Tables
Non-tabular structured data.

Tables are relatively easy to understand, but what is non-tabular structured data? As mentioned above, in the case of OpenSea's pages, if one needs to simultaneously capture the prices and numbers corresponding to NFT images in order to construct a table, this can be considered non-tabular structured data. Compared with obtaining similar elements, structured data adds a matching algorithm between different columns and a range of functions to facilitate table operation.

7.1 Capturing tabular data.

Start the Recorder in VS Code
Click data scraper in the Recorder

For example, let's capture the following table in Coingecko.

Use Ctrl+left mouse button to click on the first row and column of data in the table. Clicknium will automatically detect the capture object for the table, and prompt whether to capture the full table information.

Click Yes to see the data preview. On the preview page, you can modify the column name, change the column order, delete the column, and so on. You can also select No, and then select the desired data column by column.

7.2 Capture non-tabular structured data.

Let's use an example：Gaming chairs in Amazon website

In this example, we need to fetch the product name, price and image. The operation is similar to capturing similar elements, where two elements determine a column.

a. Capture the first cell in the first column: the first product name.

b. Capture the second cell in the first column: the second product name.

c. The first column is captured successfully, click Add column to start capturing the second column elements.

d. Capture the second column, the first element: the first product price.
......

After capturing, Clicknium will organize the data into tables.

On the preview page, you can also change the column names, and column order and select different attributes as column values.

Sample Code:
For structured data, Clicknium provides the scrape_data method, which returns text in JSON format. Not only this function supports passing in the Locator of the page flip button to achieve automatic page flip, but page flip also supports setting controls and simulating the Mouse, etc., waiting for the page to load, grabbing the number of data bars, control and timeout.

from clicknium import clicknium as cc, Locator
import pandas as pd

row = cc.scrape_data(locator.jd.phone)
df = pd.json_normalize(row)
print(df.head(10))

8. Automation Technology

By default, Clicknium will automatically select the appropriate automation technology. However, when automating desktop applications, sometimes it is necessary to choose a different automation technology, and different automation technologies are thought to have different locator attributes.

Clicknium includes automation technologies：

Note that the Locator recorded by different automation technologies has different Locator attributes and Element properties. For Element properties, you can check the documentation of the corresponding technology above and get the value of the property through the get_property method.

The following table provides a summary of the automation technologies supported by Clicknium.

Automation Technology	Scenario	Introduction
Browser Automation	Chrome/Edge/IE/Firefox etc.	Chrome DevTools Protocol/ Web elements are controlled using Clicknium browser plug-in with injected JavaScript scripts.
UIA	Windows desktop application	Based on Microsoft UI Automation, it has richer method and property information than IA, and is specially optimized for WPF.
IA	Windows desktop application	Based on Microsoft Active Accessibility (MSAA), it exposes some of the interface properties and APIs using the COM interface. Better support for software that is older in development.
Image Recognition	try with other automation tech failed	Based on Open CV, it is also the reason why the Clicknium Python SDK is relatively large
JAVA automation	automate Java GUI Application	Based on the Java Access Bridge and enhanced with AWT components and more, you need to rely on the Clicknium Java plug-in.
SAP automation	SAP WinGUI API	Based on SAP GUI Scripting.

In the Recorder, the default options can be modified to select the automation techniques you need:

9. Pack the Clicknium project as an executable file

The user of an automation script is not necessarily the Python programmer himself. When automation scripts need to be handed off to different users, there are a variety of issues that need to be addressed. It is possible that the other party does not have a Python environment, lacks a particular Python library, or has a version conflict, or even that the other party is a business person and not a programmer. Packaging scripts into human-running exe executables can be a good solution to these problems.

Packing method：

Clicknium Project
PyInstaller

9.1 Packed by Clicknium Project

Clicknium Package is Clicknium's built-in project management approach, designed to ensure that automation scripts are developed, debugged, executed and distributed in a way that guarantees the same operational results.

a. Create Clicknium Project

In VS Code, press Ctrl+Shift+P to bring up the Command Palette, select Clicknium: Create Project, and specify a path to save the project. During the project creation process, a virtual environment will be created for the project at the same time. You can see the environment information displayed next to the Python interpreter in the bottom right corner of VS Code.

Unlike a normal Python project, several files are automatically created in this Project.

app.py: Python source file
logo.icon: the Icon of the packaged executable
clicknium.yaml: the configuration file for the Clicknium Project

clicknium.yaml, which contains basic information about the project, including entry and dependencies, log paths, etc. The logo.icon can be replaced with the icon file you want.

b. run and debug Clicknium Project

Similarly, in VS Code, press Ctrl+Shift+P to bring up the Command Palette, and select Clicknium: Run Project. clicknium will run the project with the configuration in clicknium.yaml. When you run the file directly from F5, you will not see the configuration in yaml for deployment and execution.

c. Clicknium Project package

In VS Code, press Ctrl+Shift+P to bring up the Command Palette, select Clicknium: Package Project, then select a path to store the executable, then select console, if the script contains a graphical interface, please select GUI, otherwise the graphical interface may not be displayed properly. Clicknium will package the script according to the information in clicknium.yaml.

The following is an example of a clicknium.yaml file.

startUp: app
ignoreFiles: .gitignore
log:
  folder: 
  type: File
type: project
requirements:
  python: 3.9.13
  pip: 
  packages:
  - package: clicknium
    version: 
  - package: pandas
    version: 1.5.3
  locators: []

In the above example：

app:the entry point file of the Python program

log: The default log path is C:\Users{currentUser}\AppData\Local\Clicknium\Log. It can be modified to other paths. For example, the current path is .\

python: the version of the Python interpreter to be packaged into the executable file.

Clicknium will automatically scan for Python libraries used in the script and automatically add the corresponding library name and version number. If there are problems with the automatic addition, you can modify this section manually.

9.2 Package by PyInstaller

In addition to the built-in packaging method, a more common practice in Python development is to use the Pyinstaller: PyInstaller Manual.

To keep the project clean, it is still recommended to use a Python virtual environment for development. Typically, the Python virtual environment used for a project can be found by opening the settings.json file in the .vscode folder of the project file.

a. Go to the directory of the virtual environment and use pip install pyinstaller to install pyinstaller. use the pip list to see what pyinstaller is included and the specific version information.

b. Edit publish.spec

Configuration description：

Project location：{$"project app.py full path"}，input the full address of the project app.py, such as 'c:\Users\{Username}\Desktop\Sample\app.py'
Clicknium SDK address：($'clicknium .lib automation full path', 'clicknium\.lib\automation')，input Python virtual environment site-packages folder clicknium under.lib\automation，such as：binaries=[('C:\Users\{Username}\AppData\Local\Clicknium\Envs\Sample_17aea73e4ca65262\.virtualEnv\Lib\site-packages\clicknium\.lib\automation', 'clicknium\.lib\automation')]
Locator address：datas=[($'project .locator full path', '.locator')]，input the .locator address of project, such as, datas=[('c:\Users\YanZhiwei\Desktop\Sample\.locator', '.locator')]
Executable file name：name=$'project name', Fill in the package to generate exe name. You can fill in your own

The following is an example.：

# -*- mode: python ; coding: utf-8 -*-


block_cipher = None


a = Analysis(
    ['c:\Users\KayYOLO\Desktop\Sample\app.py'],
    pathex=[],
    binaries=[('C:\Users\KayYOLO\AppData\Local\Clicknium\Envs\Sample_17aea73e4ca65262\.virtualEnv\Lib\site-packages\clicknium\.lib\automation', 'clicknium\.lib\automation')],
    datas=[('c:\Users\KayYOLO\Desktop\Sample\.locator', '.locator')],
    hiddenimports=[],
    hookspath=[],
    hooksconfig={},
    runtime_hooks=[],
    excludes=[],
    win_no_prefer_redirects=False,
    win_private_assemblies=False,
    cipher=block_cipher,
    noarchive=False,
)
pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher)

exe = EXE(
    pyz,
    a.scripts,
    a.binaries,
    a.zipfiles,
    a.datas,
    [],
    name='Sample',
    debug=False,
    bootloader_ignore_signals=False,
    strip=False,
    upx=True,
    upx_exclude=[],
    runtime_tmpdir=None,
    console=True,
    disable_windowed_traceback=False,
    argv_emulation=False,
    target_arch=None,
    codesign_identity=None,
    entitlements_file=None
)

10. Frequently Asked Questions：

5.1 How to launch a Windows application

You can launch desktop applications using Python's subprocess module

import subprocess

process_notpad = subprocess.Popen("notepad")
process_feishu = subprocess.Popen("C:\Users\kay\AppData\Local\Feishu\Feishu.exe")

10.2 How to set a timer to run

a. Search for Task Scheduler in the taskbar

b. Create a Task or Create a basic Task

c. Task Config：

Task name
Trigger：Select trigger conditions, daily, weekly or monthly, at startup, at login, etc.
Action： Select Launcher, Program or Script: Fill in the address of python.exe. Add parameter: the address of the automation script.

10.3 For the same website, if you open multiple browsers to log in to different accounts

With the browser automation extension, attach method is the choice. If you don't need to mouse and keyboard simulation, it is recommended to use Chrome and Edge browsers to call the CDP interface and bind the tab object to the corresponding browser tab. To make web browser instances not share cookies between them and to be able to log into different accounts on the same website, different User profiles can be used.

According to open method method：

def open(
        self,
        url: str,
        is_maximize: bool = True,
        is_wait_complete: bool = True,
        userdata_folder_mode: Literal["automatic", "default", "custom"] = WebUserDataMode.Automatic,
        userdata_folder_path: str = "",
        timeout: int = 30
    ) -> BrowserTab

set userdata_folder_mode to 'custom', and pass different folder paths to userdata_folder_path，as different addresses to store the user profile. For different browser tab, use differnet path, instant multi-account login is available.

10.4 Why sometimes the script will be stuck in the main for a long time? How to check the log？

Most scripts get stuck because they cannot find the UI element corresponding to the Locator. Clicknium has a default timeout of 30 seconds for locator operations. You can check the API documentation for the timeout time of the method. If you get stuck, first use Validation to make sure that the Locator can recognize the UI element.
If it didn't work, use Recapture&Compare to check the difference for the current UI element attribute and the old locator attribute.
The default path to the logs：C:\Users{currentUser}\AppData\Local\Clicknium\Log

10.5 Why did the Locator fail after a while

There are many reasons why a locator may fail, such as dynamic loading, version updates, and context changes. First, you can use the Recapture&Compare function on the locator page to re-record and compare the differences between the old and new locators. Different properties will be highlighted.

To make the Locator more stable, it is recommended to use wildcards, regular expressions, parameterized Locator, and other features to improve the Locator's robustness.

10.6 Why Mouse click and type text has no effect

Toggles between the system call method and the simulated mouse method. In some cases, when recording a button, it may record the logo inside the button or the outer border. For system calls, only the button has the click property. Some input boxes also have a click or focus state to be activated, so you can choose to click or focus parameters before input. Since there is no return value after the system call is triggered, Clicknium cannot throw an error. In such cases, it is recommended to click the button in a simulated mouse way.

How we migrate Clicknium to Chrome Manifest V3

kayYOLO — Thu, 03 Nov 2022 03:07:18 +0000

Introduction

We started a project called Clicknium this year, a Python automation library for browser and desktop applications. It depends on the Clicknium browser extension to control the browser, and simulate mouse and keyboard actions.

Since January 2022, adding new extensions based on Manifest V2 to the Chrome Web Store has been forbidden. All extensions on Manifest V2 will stop working in early 2023, even those that were added to Chrome Web Store earlier.

We have no choice but to migrate to V3. After we review our code and migration checklist,
the most significant impact on Clicknium is that running remote code will be forbidden in Manifest V3.
It asked to include all logic in the extension's package to review the behavior of the extension better.

How does Clicknium run remote code in Manifest V3? Here is a brief introduction.

How Clicknium runs remote code in V2

This is a simple Clicknium automation case, which executes a JS script on the Bing page to set the value of the search box to "Clicknium", and return "success".

from clicknium import clicknium as cc, locator
import os,sys

def main():
    tab = cc.chrome.open("bing.com")
    ele = tab.find_element(locator.bing.cn.search_sb_form_q)
    result = ele.execute_js("function SetText(st){_context$.currentElement.value = st; return \"success\"}", "SetText(\"Clicknium\")")
    print(result)
if __name__ == "__main__":
    main()

In Manifest V2:

In the background: Through chrome.runtime.sendMessage sends the received string to the content.
In the content: Using "new Function(jsCode)" to convert the received string to Javascript's Function,

const innerFunction = new Function('_context$', jsCode);
const ret = innerFunction({
    currentElement: targetElement,
    tabId: msg.tabId
});

In Manifest V3, Function and eval is not available. So the main problem is that how to convert string to Javascript's Function.

How Clicknium run remote code in V3

There are two approaches to achieving it.

Solution 1: Use pop up window to imitate the background scripts/pages.

In Manifest V3, Using Service worker replace background scripts/pages. The window object can't be used in the Service worker.
But we can open a popup window in the Service worker to imitate background scripts/pages.

Step 1 : Open a popup window in the Service worker.

    // Get the URL of the popup window.
    const backgroundUrl = chrome.runtime.getURL("backgroundPage.html");

    await chrome.windows.create({
        type: "popup",
        state: "minimized",
        url: backgroundUrl,
    });

Step 2 : In the popup window, when received a message, using chrome.debugger.sendCommand sends to the target of the popup window. And then, get Function from the window object.

    // Get the target of the popup window
    const backgroundUrl = chrome.runtime.getURL("backgroundPage.html");
    const targets = await chrome.debugger.getTargets();
    let target;
    for (const t of targets) {
      if (target.url === backgroundUrl) {
        target = t;
      }
    }

    // Wrap up the Javascript's string, register to the window object
    const jsCodeWrapper = `window["evalFunction"] = function () {
      ${jsCode};
    };`;

    // Attach to the popup window. And send expression.
    await chrome.debugger.attach(target, "1.2");
    await chrome.debugger.sendCommand(target, "Runtime.evaluate", {
      expression: jsCodeWrapper,
    });

    // Get the function just registered in the window
    const wrapperFunc = window["evalFunction"];

Step 3: Using chrome.scripting.executeScript to run Function in content.

    await chrome.scripting.executeScript({
      target: { tabId: msg.tabId, frameIds: [msg.frameId] },
      func:wrapperFunc,
    });

Advantage:

Remote code works in content and background.
Support to cross-domain iFrame.

Disadvantage:

Need to open a popup window.

Solution 2: In Service worker, using chrome.debugger.sendCommand sends to content.

    interface CDPRuntimeEvaluateResult {
        exceptionDetails?: { exception?: { description?: string } };

        result: { value: unknown };
    }

    await chrome.debugger.attach({tabId: msg.tabId}, "1.2");

    const result = (await chrome.debugger.sendCommand(
      { tabId: msg.tabId },
      "Runtime.evaluate",
      {
        expression: js,
      }
    )) as CDPRuntimeEvaluateResult;

Advantage:

No need to open a popup window.

Disadvantage:

Not support for cross-domain iFrame. Cross-domain iFrame's target isn't in the list of available debug targets, which uses the chrome.debugger.getTargets to get.
Remote code can't work in the background.

Summary

As cross-domain iFrame is a critical feature we need, Clicknium choosesSolution 1: use popup window to call chrome.debugger.sendCommand.

Manifest V3 has been supported by Clicknium since version 0.1.10.
You can try it in
pypi website and Clicknium VS Code Extension.

One thing that should be noticed is that the Clicknium browser extension has to work with the Clicknium Python SDK. If we publish the browser extension to Chrome Web Store, all the existing extensions will auto-upgrade to the latest version, and it can not work with the SDK before V0.1.0. So we have to pull it off shelves and wait for a period. If you want to try the M3 version, please upgrade to the latest Python SDK.

[Python]Facebook Scraper via Clicknium

kayYOLO — Thu, 13 Oct 2022 14:29:39 +0000

Facebook Scraper

This is a sample to scrape Facebook posts using Clicknium.

Preparation

Python 3.7+
Windows 7 SP1+
Chrome browser
VS Code
Clicknium
Clicknium Chrome extension

Scrape Facebook posts

We will scrape the post of the Facebook company page as an example.

Create a Python project

Create a Python file, for example, sample.py, under a project folder.

Show Locators under the VS Code Explorer:

Capture locator

A locator is a tool that targets the UI elements.

Login and open the Facebook company page: https://www.facebook.com/facebook
Click the Capture button in VS Code.

Click similar elements This feature lets you get all the posts on the page that have the same structure.

Use Ctrl + Click to capture the first post words on Facebook:

Capture the second post in the same way:

You will see there are five elements matched. Click the save button and finish.

Get the text via Locator:

To get the locator targets, we can use find_element function. In our scenario, we need to get multiple posts so we can use find_elements function to get a result array.

UIs = cc.find_elements(locator.facebook.posts)

In the Python code, we can use Locator. to use the locators we captured using Clicknium in this project. If there is a need to use the same Locator across projects, you can make the locator store into a cloud locator store, and you can reference it anywhere.

When we get the UI elements, we need to find an element property that can contain the text info. Check the Web elements property. The property innertext is what we need.

    uis = cc.find_elements(locator.facebook.posts)
    for ui in uis:
        text = ui.get_property("innertext")

we can print the text to check if it works or not.

If it doesn't work, we need to check the locator page to tune the property. The Locator uses the identity UI elements.

On the above page, you can do some quick validation and action to check if the Locator can work or not. And you can also select and modify which attribute you want to use to locate the UI elements.

Go to the next page

The Facebook content will be loaded when you scroll down the page. So if we capture once, we can't get all the information. So we have to capture each page. If we mimic the scroll action of the mouse, it would be hard to control. So the best choice would be to use the PageDown button in the keyboard. The send_hotkey function can do it easily. we can find the Code for PageDown is {PGDN}.

cc.send_hotkey("{PGDN}")

We can use a while loop the get all the posts. Since the multiple capture would get the same post for times, we can use a dictionary to store the post and use ancestorid as the key.

Source code

GitHub

How to attach to existing Browsers in Web Automation

kayYOLO — Tue, 11 Oct 2022 12:52:24 +0000

Overview

For web automation, there are common scenarios containing end-to-end automation without human intervention, headless browser automation, and partial automation.

Partial automation means that not the entire business process is automated, and humans may operate some in advance due to security concerns, complex operations which are not repetitive but time-consuming, etc. In this automation case, the browser is opened, the human will do the relevant preparation work, and then the automation script will be executed for the repetitive part.

In this case, attaching the existing browser is necessary for the automation script to handle the same browser which human works on. We are introducing the browser attaching method for different automation libraries here, containing Selenium, Playwright, and Clicknium.

Selenium attaches to the existing browser

Requirements

Download ChromeDriver from here. You need to select the same version of chromedriver.exe with your installed chrome browser, which you can see it by chrome://version in chrome browser.
Place the chromedriver.exe in the same folder as Chrome.exe. For example, if your Chrome browser file path is C:\Program Files\Google\Chrome\Application\chrome.exe, then the chromedriver.exe could be accessed with C:\Program Files\Google\Chrome\Application\chromedriver.exe.
The browser that needs to be attached should be started with argument --remote-debugging-port, and you also need to operate on this browser. The command looks like this: "C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=8888
Notice that if there are other Chrome browsers running in the same time, it is better to close the others and keep the only one that you want to attach later, or the Selenium script might not successfully attach to the right browser, or the page that you want to automate may get refreshed when attaching. ## Code The idea is to start a Chrome instance by specifying the option of debugger_address, the port in the address is what we specified in the command before. And then, get to the page where you want to run the Selenium automation script after attaching.

from selenium import webdriver
from selenium.webdriver import ChromeOptions
from selenium.webdriver.chrome.service import Service
options = ChromeOptions()
options.debugger_address = "127.0.0.1:" + '8888'
browser = webdriver.Chrome(service=Service(executable_path="C:\\Program Files\\Google\\Chrome\\Application\\chromedriver.exe"), options=options)
browser.get("https://contoso.com/u/0/#in?compose=new")

The playwright attaches to the existing browser

Requirements

Run playwright install to init the playwright prerequisites
The browser that needs to be attached should be started with argument --remote-debugging-port , and you also need to operate on this browser. The command looks like this: "C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=8888
Notice that if there are other Chrome browsers running in the same time, it is better to close the others and keep the only one that you want to attach later. And if there are multiple windows opened from the browser started by the above command, it is better to keep one page with the Url that you want to automate to avoid attaching to a non-target page.

Code

The playwright provides two ways to attach to existing browsers: connect and connect_over_cdp. The connect needs to work with launchServer in Node.js, so we will use connect_over_cdp in Python to manually start the Chrome browser. The code is as below:

from playwright.sync_api import sync_playwright
with sync_playwright() as p:
    browser = p.chromium.connect_over_cdp("http://localhost:8888")
    default_context = browser.contexts[0]
    page = default_context.pages[0]

Clicknium attaches to exsiting browser

Requirements

Install Clicknium VSCode Extension and Python module pip install clicknium. More about Clicknium.
Install the Clicknium browser extension. There are two ways of doing this:
Install via Python:

import clicknium as cc
cc.chrome.extension.install()
Install via VSCode extension UI:

Notice that the browser extension needs to be enabled after installation. ## Code Clicknium provides two ways to attach existing browsers, attach and attach_by_title_url. The attach is to attach a browser by a web element specified in the target browser. The attach_by_title_url is to attach to a target browser by title or Url. This is more flexible if the target page element can be dynamically changed. The code is as below: (we may use wildcard * if part of the Url may change)

from clicknium import clicknium as cc, locator
browser = cc.chrome.attach_by_title_url(ulr='https://contoso.com/u/0/in?id=*')
#browser.find_element(locator.hundun.img)

Compare

It is not easy to make attaching to the browser stable. If you have any suggestions on this, please feel free to leave a comment. Thanks.

[Python]Telegram Automation

kayYOLO — Sat, 08 Oct 2022 12:16:23 +0000

Telegram Automation

Overview

It is a sample of using Python code to control the Telegram Windows client and send messages to contacts and the Telegram channel.

Step by Step

Set up Clicknium
- Install VS Code Clicknium Extension
- Use the extension to install Clicknium Python SDK
Log in to your Telegram client.
Capture the search bar
- Click Capture button
- Use Ctrl + Click to capture search bar
- Get the search bar locator
- Validate the locator The validation would be failed. The most simple way to debug this issue, we can use a new feature provided by Clicknium: Recapture & Compare After recapturing the search bar, we can get the compared model to check the difference between the old and new. It's easy to get the difference between the old and new is the name attribute. There is a number that follows the word "Telegram." We can use the wildcard to fix it by changing the number 383784 to * and saving the locator. Validate again, and it will succeed.
Set text into the search bar
Include the Clicknium package into your Python code and set the contacts name into the search bar using Clicknium API.

from clicknium import clicknium as cc, locator

cc.find_element(locator.telegram.contact_search).set_text("DK H")

Select the contacts by pressing the Enter key. More codes of the key can be found at the page

    cc.send_hotkey("{ENTER}")

Capture the message text input and send messages button using the same way as step 3.
Code the workflow into Python. You can check the sample in the sample.py file.

Source code

Github: https://github.com/automation9417/TelegramAutomation

Generate reliable locators for web testing or data scrapy

kayYOLO — Tue, 27 Sep 2022 11:14:14 +0000

Generate reliable locators for web testing or data scrapy

Background

Suppose you do any Web UI testing automation or web data scrapy. In that case, you probably spend a large chunk of development time finding elements on a web page, such as input, checkbox, submit button, table, and divs. It can be challenging to locate the correct elements, especially when they lack unique attributes, such as id, name, and class names; it is even worse if the attributes are dynamic. For example, the id is always changed when opening the page.

Clicknium supplies an automation library and development tool, and it can improve developer efficiency and automatically generate locators for elements on any UI (web, desktop application, etc.). The auto-generated locator is a good choice for many general cases.
However, based on some of the scenarios described above, the attributes used for positioning may be dynamic, which requires users to modify the locator. The Clicknium development tools provide multiple functions to help developers easily pick which attributes are used to locate UI elements and to make the final locator more stable.

If you are unfamiliar with Clicknium, you can try the Clicknium Visual Studio Code Extension first.

Recapture and Compare

Recapture & Compare allows you to recode the same UI element, create a new location, and present the user with a comparable view.
The user may easily distinguish the dynamic attributes by comparing the new locator to the prior one. The chosen list should not include these dynamic attributes.

Many websites today include dynamic pages, which means that when a user opens the same page, different material is shown. Locating the element will be challenging in this scenario. The element's locator was successfully found the last time, and it might fail next time. Use the "Recapture and Compare" feature to rapidly decide which properties on the locator should be used in this situation.

The example below demonstrates how to take advantage of this functionality.

Case background: The user wants to download the first video cover image from the YouTube main page.

It is simple to generate the locator like the one below using Clicknium Recorder:
When you validate the location after refreshing the web page, it may fail because the image has changed.

Let us use 'Recapture and Compare' to recapture the first image again.

You will get the following view:

As you can see from the comparative mode, the values of the attributes "src" and "sInfo" have changed, so we should disregard them and choose different attributes.

You can uncheck "src" and check "cssSelector" in this situation, then save the locator.

Compared to the old locator, the new one ought to be more reliable.

Use custom web attributes

From Locator and Web Automation, you know which attributes Clicknium used to locate web element, such as "tag", "id", "name" and so forth.

A web element may occasionally include additional attributes, some of which are significant and can be used to identify the web element.

The example below demonstrates how to take advantage of this functionality.

Sample 1: Azure DevOps new work item page

We want to locate the 'Discussion' input area.

It is simple to generate the locator like the one below using Clicknium Recorder:

In this scenario, the "ancestorId" of the produced locator is dynamic; if you create a work item again, the "ancestorId" may change.
Given that Clicknium returns the values for all of the web element's attributes, choosing "aria-label" and excluding "ancestorId" will make the locator more dependable for this sample.

In order to be flexible, you should alter the "title" to "New Bug *: - Boards".
Please refer to wildcard locator.

Conclusion

Clicknium provides several ways to make locator more robust:

Automatically generate locator, feasible for many general cases.
Supply 'Recapture&Compare' tool to help developers to choose stable attributes
Use custom attributes of web UI element

OpenSea NFT Scraper

kayYOLO — Fri, 23 Sep 2022 11:00:48 +0000

Overview

OpenSea has a lot of NFT pictures. It's a good example of learning how to scrape. Compared to other typical static websites, pictures in OpenSea are dynamic loading along with the endless scroll. Pictures are loading if they are on the screen and will be gone
when they are out of the screen.

Clicknium

Clicknium is a Python automation library and very easy to do this kind of scraping work. It's more than a Python library but also provides a smooth and interactive coding experience with VS Code plugins and browser plugins.

Requirement

Windows7 SP1+
Python 3.7+
VS Code

How to

Let's scrape Okay Bear as an example.

Picture info

Open Okay Bear's page with Chrome.
Press F12 to open the DevTools, click the mouse icon, and then click the NFT picture. In the following pic, we can see:
Image alt can be used as picture name
Picture src and srcset with different sizes.

Capture Pictures

Capturing similar elements is quite easy for this scenario:

Click the capture button in VS code extension to start the recorder.
Click Similar elements .
Ctrl + Click on the first pic and then capture the second.

You can also use this way to capture the price.

Handle scroll to load pictures

Clicknium provides API to control the mouse and keyboard. In this Scenario, Page-down is the easiest way. More about HotKey

clicknium.send_hotkey("{PGDN}")

Download picture

use requests(pip install requests)

Source code:

Github: OpenSeaScraper

import requests
from time import sleep
from clicknium import clicknium as cc, locator
def main():
    cc.chrome.open("https://opensea.io/collection/okay-bears")
    nftDic = {}

    while True:
        newDic = {}
        finish = True
        sleep(4)
        nfts = cc.find_elements(locator.sample.opensea.img_okay_bear)
        for nft in nfts:
            src = nft.get_property("src")
            name = nft.get_property("alt")
            if name not in nftDic:
                newDic[name]=src
                finish = False

        for key,value in newDic.items():
            r = requests.get(value)
            print("Downloading:"+ key)
            imagePath = ".\\image\\" + key + ".jpg"
            with open(imagePath,'wb') as f:
                f.write(r.content)

        nftDic = nftDic | newDic
        if finish:
            break
        else:
            print("next page")
            cc.send_hotkey("{PGDN}")

if __name__ == "__main__":
    main()

More sample:
10 lines of Python code to upload a video to TikTok, Instagram, Twitter

A Spotify robot

[Python]Keep MS Teams status active

kayYOLO — Thu, 15 Sep 2022 11:36:44 +0000

Microsoft Teams will auto-offline without action from your mouse and keyboard for a while. So we can use Python to control the mouse and keyboard to keep moving.
Clicknium is a Python Library that can control your mouse and keyboard.

install

pip install clicknium

Python script

from time import sleep
import math
from clicknium import clicknium as cc
def circle():
    a,b = cc.mouse.position()
    w = 20  
    m = (2*math.pi)/w 
    r = 200      

    while 1:    
        for i in range(0, w+1):
            x = int(a+r*math.sin(m*i))  
            y = int(b+r*math.cos(m*i))
            cc.mouse.move(x,y)
            sleep(0.2)

if __name__ == "__main__":
    circle()

The above code will keep moving your mouse along a circle. You can stop it by `Ctrl+C.

There are a lot of things that you can use Clicknium to achieve and it is quite easy. It can control the mouse and keyboard, automate desktop applications and web browsers. Follow the Clicknium quick start and turn your routine work into an automation workflow.

A Spotify robot

kayYOLO — Fri, 26 Aug 2022 13:41:57 +0000

Spotify robot

I want to scrape a Spotify playlist from a web page, at first I try to use selenium, but it is not obtaining the full list, as the song list is dynamically loaded, I must scroll down to the bottom, then it can load all songs.

Finally, I find one Visual Studio Code automation development tool, clicknium (it should be new in 2022), it is really easy to use and one click to locate the UI element.
I just need to record two locators and write several lines of python code, the job is accomplished.

As the list is dynamically loaded, I use an iterate index to search each item, as clicknium can automatically scroll the item into view, it will trigger to load new items, so I don't need to take care of the dynamic loading.

I use a parametric locator for title and author

As one item may have multiple authors, so I use the following code to get multiple author's names and links:

    element = tab.find_element(locator.chrome.open.div_author, {'index':index})
    authers = element.children
    for item in element.children:
        artist.append(item.get_text())
        link.append(item.get_property('href'))

If you want to try Clicknium, you can get started with the video on clicknium official site.

[Python]Control MS paint to draw a pixel style logo automatically

kayYOLO — Fri, 26 Aug 2022 10:32:00 +0000

From:

To:

Firt of all, perpare a simple piture.
Install Python package:
- pip install opencv-python
- pip install clicknium > opencv-python: CV library. Load image and convert to a pixel style graph. > clicknium: Automation library. Lanch MS paint and control the mouse to pain the picture.
Search and install clicknium in VS Code extension market.
Get the source code
git clone https://github.com/automation9417/mspaint-draw-img.git
Use VS Code to open the souce code folder and press F5to run. Use Shift+F5 to stop.

You can also build your own automation task using Clicknium. This tool is very easy to use. You can use the Recorder to Capture UI element for both Windows applications and web browser. It will auto-generate the locator to help you to locate and control the UI elements.

The Python script can also be package to a executable file and send to other people. Draw a something special or type some words that you want to speak to the one.

More detial please check the doc

10 lines of Python code to upload a video to TikTok, Instagram, Twitter

kayYOLO — Wed, 24 Aug 2022 05:33:34 +0000

First, let's look at the running video.

upload video to Tiktok, Instagram, Twitter

Material Prepared

An mp4 video file, because twitter has a limit on the length of uploaded videos, a shortened video file has been prepared.
A cover image for Instagram, TikTok uses the first frame of the video as cover.

Run Python Code

follow clicknium getting started to set up develop environment.
clone sample repo.

git clone https://github.com/automation9417/automation-samples.git

open the folder 'UploadVideo' in Visual Studio Code.
open sample.py in Visual Studio Code.
open the chrome browser (guarantee that there is only one chrome window, explained later), open 3 tabs respectively, open and login TikTok, Twitter and Instagram.
press F5 to debug the sample or press CTRL+F5 to run sample. The video that comes with the sample will be uploaded, and you can also modify code to your own video content:

caption = 'Clicknium introduction'
cover_image = os.path.join(os.getcwd(), "media", "logo.png")
video_file = os.path.join(os.getcwd(), "media", "clicknium_introduction.mp4")
upload_tiktok.upload(caption, video_file)
short_video_file = os.path.join(os.getcwd(), "media", "short_introduction.mp4")
upload_twitter.upload(caption, short_video_file)
upload_instagram.upload(caption, cover_image, video_file)

Implementation

Here is an example of uploading a video to Instagram, uploading a video to TikTok and Twitter is a similar process.

Get the open browser tab by attaching the browser, and then navigate to the Instagram Home page.

tab = cc.chrome.attach_by_title_url(url="https://*instagram.com/*")
tab.goto("https://www.instagram.com/")

Click the 'add post' button, all web page elements here are recorded by Clicknium Recorder.

tab.find_element(locator.chrome.instagram.svg_add_post).click()

Click the 'Select from computer' button to pop up a file selection dialog box. Due to browser security restrictions, click here by simulating a mouse.

tab.find_element(locator.chrome.instagram.button_select_file).click(by='mouse-emulation')

Using Clicknium desktop automation to select video file and click open button.

ui(locator.chrome.edit_file).set_text(video_file, by='set-text')
ui(locator.chrome.button_open).click(by='control-invocation')

A special note here is that I changed the recorded locator 'edit_file' and 'button_open':

Changed the window matching Name to the wildcard *, because uploading videos to TikTok and Twitter also requires the operation of the file selection dialog. In order to avoid repeated recording, the wildcard is used to match the window to achieve reuse. Therefore, the previous steps require only one Chrome browser window to ensure that the window can be correctly matched.

Click the 'next' button, then select the cover image, the code is similar to the above.
Enter the text you want to send, then click 'Share' button.

tab.find_element(locator.chrome.instagram.textarea).set_text(caption)
tab.find_element(locator.chrome.instagram.button_share).click()

Wait for the upload to succeed, the time required varies according to the size of the video.

tab.wait_appear(locator.chrome.instagram.h2_yourposthasbeenshared, wait_timeout=120)

Uploading a video to Instagram requires a few more steps to upload a cover image. The following is the complete code for uploading a video to Twitter. It only takes 9 lines of code.

tab = cc.chrome.attach_by_title_url(url="https://*twitter.com/*")
tab.goto("https://twitter.com/compose/tweet")
tab.find_element(locator.chrome.twitter.div).set_focus()
cc.send_text(tweet)
tab.find_element(locator.chrome.twitter.svg).click()
ui(locator.chrome.edit_file).set_text(video_file, by='set-text')
ui(locator.chrome.button_open).click(by='control-invocation')
tab.wait_appear(locator.chrome.twitter.video)
tab.find_element(locator.chrome.twitter.span_tweet).click()

You are welcome to comment on the needs of uploading videos to other video platforms, and I can provide more sample code.

GUI automation under the hood

kayYOLO — Tue, 23 Aug 2022 12:34:00 +0000

Clicknium provides the automation technology in a variety of scenarios with different forms, such as SAP automation, Java automation, Windows desktop automation and image automation. I would like to share the automation technology in Clicknium.

Introduction

Regarding software automation technology, it was mainly used for software testing in the very beginning, especially for UI automation testing. Many system software designs take the accessibility of disabled users into account. See windows accessibility. In recent years, RPA (Robotic Process Automation) has been accepted by more and more enterprises. RPA plays a great role in multiple business field scenarios to optimize the process, reduce costs and increase efficiency, and quickly improve the level of informatization. RPA products generally include automated operation components for various software, data processing components, process logic control components, etc., among which UI automation components are the mostly used.

Windows UI Automation Technology

Windows applications include a functional design: Accessibility, which allows more users to operate the software. For example, allow the blind users to read the screen through their voice, etc. The UI automation technology provided by Microsoft can access the control elements of the windows application. The main technologies are as follows:

UIA (UI Automation) UI Automation provides programmatic access to information about the user interface (UI) for Microsoft Windows, enabling assistive technology products (such as screen readers) to provide information about the UI to end users using means other than standard input and Manipulate UI. UI Automation also enables automated test scripts to interact with the UI.
MSAA（Microsoft Active Accessibility） Microsoft Active Accessibility was the application accessibility solution from Microsoft earlier.

	UIA	MSAA
Programing Languages	Written in managed code, client applications are most easily programmed using C#	based on the Component Object Model (COM) with support for dual interfaces.
Windows Presentation Foundation	full support	WPF do not contain native support for MSAA
Servers and Clients	a core service between the server (called a provider) and the client, providing more information to the client	servers and clients communicate directly, largely through the server's implementation of IAccessible
Security	UIA removes the need of providers to call through to other provider code. The UI Automation core service does all the necessary aggregation.	Some IAccessible customization scenarios require wrapping a base IAccessible and calling through to it. There is a potential security risk, since a partially trusted component should not be an intermediary on a code path.

If you are interested in Windows automation technology, you can directly go to Microsoft's official technical documentation.

Java Access Bridge (JAB)

Java Access Bridge is a technology that exposes the Java Accessibility API in a Microsoft Windows DLL, enabling Java applications and applets that implement the Java Accessibility API to be visible to assistive technologies on Microsoft Windows systems. Java Accessibility API is part of Java Accessibility Utilities, a set of utility classes that help assistive technologies provide access to GUI toolkits that implement the Java Accessibility API.

To learn more about JAB, please visit oracle

Web Automation

Web automation mainly depends on the type of browsers, and currently, there are two main categories:

Internet Explorer(IE)

Through javascript injection based on the Component Object Model (COM).
Due to different versions of IE, it is still a technical issue to make multiple versions of IE compatible.

Chromium-based browsers

There are generally three ways:

Selenium webdriver To learn more about webdriver, please visit WebDriver, Chrome webdriver.
Chrome devtool protocol(CDP) Since CDP is used in many tools, you can refer to awesome-chrome-devtools. Related to web automation testing, they are Puppeteer and Playwright. The playwright's python code to start the Chrome browser is as below:

browser = p.chromium.launch(channel="chrome", headless=False)
context = browser.new_context()
# Open new page
page = context.new_page()
# Go to https://www.google.com/
page.goto("https://www.google.com/")

With ProcessExplorer, you can see that Chrome starts the parameters as follows.

"C:\Program Files\Google\Chrome\Application\chrome.exe" --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,AcceptCHFrame,AutoExpandDetailsElement --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=C:\Users\test\AppData\Local\Temp\playwright_chromiumdev_profile-V3LZUg **--remote-debugging-pipe** --no-startup-window

According to the chrome startup parameter description:

--remote-debugging-pipe: Enables remote debug over stdio pipes [in=3, out=4]. Optionally, specifies the format for the protocol messages, can be either "JSON" (the default) or "CBOR"

For the way to use CDP, you can get started from this link. More Chrome startup parameters can be found in the official documentation or here.

Native messaging Extensions and apps can exchange messages with native applications by an API similar to the other message passing APIs. Native applications that support this feature must register a native messaging host knowing how to communicate with the extension. Chrome starts the host in a separate process and communicates with it using standard input and standard output streams. More about Native Messaging, please visit this link.

SAP Automation

As SAP provides script automation officially, most RPA products automate SAP applications based on their own automation technology.

Automated supplementary technology

In addition to the above native automation technologies, the following automation technologies are generally used to achieve hybrid automation.

Image recognition, shortcut keys
OCR
Screen word picking
Computer vision
Windows Native API

What Clicknium Does

With the automation technologies above, Clicknium offers development tools that allow users to:

Capture UI elements and store them as locators or selectors.
Automate all kinds of applications with a single Python API.
Higher efficiency in writing code with locator IntelliSense. In addition to the automation technologies above, we also add other enhancements:
Combine various automation technologies to increase robustness.
Find the UI component with a fast searching algorithm.
Algorithm for generating locators.

You can visit Clicknium Visual Studio Code Extension change log to see more functions.