What is Sikuli?
Sikuli is an automation tool built in Java.
SikuliX can be used by software engineers, developers, test engineers, or anyone.
It allows you to automate anything on your screen on Windows, Mac, and some Linux operating systems.
Sikuli uses OpenCV, image recognition technology, to identify GUI components.
SikuliX comes with basic text recognition that can be used to search text in images. This feature is Tesseract.
SikuliX has an IDE, an integrated development environment, for easy WYSIWYG automation.
SikuliX also has the ability implement scripting via:
Python language level 2.7 (supported by Jython)
running RobotFramework text-scripts is supported (see docs)
Ruby language levels 1.9 and 2.0 (supported by JRuby)
JavaScript (supported by the Java Scripting Engine)
We will focus on using the IDE in this article.
Installation.
First, you’ll need to download the IDE, for your particular system, from Sikuli’s launch page:
Download SikuliX here.
If you do not have Oracle Java JDK (Java Development Kit) you will need to download and install it.
Download Oracle Java JDK here.
Similarly, you can use OpenJDK if your system requires it or is a preference.
Download OpenJDK here.
*You will need to download JDK or OpenJDK 8 or better.
To open SikuliX double-click on the jar
Or type in the following in the command prompt:
java -jar sikulix.jar -v
*If you find you cannot get this command to work, make sure you have the java jdk in your PATH.
Getting Started with SikuliX.
This is the SikuliX IDE:
Idea behind SikuliX is to have a simple script that uses image recognition.
As you can see on the IDE you have two main areas.
The main scripting area
The messages area which you can think of as a console.
Below the header you will see the follow buttons:
Take screenshot - takes a screenshot of a desired image of the screen
Insert Image - this button will insert a image you’ve already taken into your script
Region - This will take a shot of a region of the screen.
Location - This will allow you to click on the screen and get the precise location (X,Y).
Offset - This will allow you to get a new location which is dx and dy pixels away horizontally and vertically from the current location.
The SikuliX script commands are:
click(): The click command allows you to click on anything you see on your screen. Simply insert your image or screenshot with the click method ex. click(“image”).
type(): Use this command anytime you wish to type on the screen ex. type(“Hello World”). This is useful for textboxes and forms.
double-click(): Similar to the click command but clicks twice instead of once.
right-click(): Use this command to make use of right-click features of your mouse.
if(),elif(),else(): Use these commands if you need logic within your script. I will show an example later.
There are many other options and commands you can use. For our purposes, these 7 commands will suffice. For more, see the documentation here
Here is a simple SikuliX script:
This script uses only two commands and four lines of script. As you can see only two images were used for this demonstration.
First, with Google Chrome open, take a screenshot of the Google search bar area.
It will place the image in your script. You’ll need to type “click(image)” around it or simply type click() and then place the cursor between the parenthese, then take the screenshot. I prefer the second way.
We “click” in this area in order to make sure our cursor is there.
Next we type our search into the search bar using the “type()” command
type(“How to search Google”)
Next we use the “type()” command again to type enter.
type(Key.ENTER);
Finally we use the click() command, in the same manner ass before, on a web page found in our search.
In the end your script should look like this:
At this point, with your Google Chrome open, you can hit the run button.
The end result should be the following
Page:
Scripts can be as simple or complicated as you would like.
Use the if(), efif(), else commands when you need to use logic. An example would be checking to see if something exists on the screen:
I would encourage you to read the documentation for a complete list of command usage.
Enjoy automating!
Top comments (1)
Tips to improve your blog posts:
1) Table of contents : every blog post you create should have a table of contents. This gives the reader an easy way to navigate your posts
2) Use markdown : all of your posts should be using Markdown. This allows for a more fluent and enjoyable reading experience through headers, subtitles and lists.