DEV Community: Rasukarusan

Shellnium: Simple Selnium WebDriver for Bash

Rasukarusan — Thu, 25 Feb 2021 06:06:30 +0000

Selenium can be used in many languages

Selenium, as we all know, is a tool for automated browser operations.
Selenium uses a WebDriver to perform automatic browser operations, and there are WebDrivers for various languages such as php, python, and javascript.

For example, if you want to use php, you can use facebook-webdriver like this.

// run chromedriver
$driver = RemoteWebDriver::create($host, DesiredCapabilities::chrome());
// open the URL
$driver->get('https://www.google.co.jp/');

For javascript, you can use selenium-webdriver to do this.

// run chromedriver
const capabilities = Capabilities.chrome()
const driver = await new Builder().withCapabilities(capabilities).build()
// open the URL
await driver.get('https://www.google.co.jp/')

It can be written in a similar way in almost any language, so if you have experience with Selenium in any language, you should be able to write it easily in other languages.

Yes, except for ShellScript.

Why isn't there a WebDriver for Bash or Zsh?

As far as I can see, there is no WebDriver for ShellScript, so if you want to try Selenium with Shell, you'll have a hard time.
You need to do npm install, composer install, etc. to get the environment ready.
I thought it would be easier for me to automate browser operations, but that's not the case.

Selenium for Shell Hackers - Shellnium

So, this is for those who want to use ShellScript to easily perform automatic browser operations.
As shown below, automatic browser operation can be achieved with ShellScript alone. All you need is bash or zsh.

How to perform automatic browser operations with Shell

The flow of automatic browser operation is as follows. All you have to do is POST JSON to the browser (driver) with curl.

Browser Launch
URL transition
Keyboard input

one by one to get a feel for it.

0. Launch chromedriver

First of all, you need to run chromedriver. if you don't have chromedriver, just brew install chromedriver. (or download it directly from the official website).

$ chromedriver
Starting ChromeDriver 2.40.565386 (45a059dc425e08165f9a10324bd1380cc13ca363) on port 9515
Only local connections are allowed.

Then chromedriver wait at http://localhost:9515.

1. Launching the Browser & Getting the Session ID

Let's start the browser from Shell.

curl -X POST -H 'Content-Type: application/json' \
     -d '{"desiredCapabilities": { "browserName": "chrome" }}' \
     http://localhost:9515/session

That's it, it's that simple. If you run it, you will see the browser start up.
In order to send commands to this browser in the future, we will need to specify the session and send the commands, so we will need to obtain the session ID.

The sessionID will be returned as JSON in the above curl response, so extract the sessionId in it. Since we are using ShellScript, it will be easier to use jq. Of course, using grep is also good.

$ curl ... http://localhost:9515/session | jq -r '.sessionId'
# Get the sessionID
e1c3d895e7c7d42383c110e5e4cd7bbf

2. URL transition

curl -X POST -H 'Content-Type: application/json' \
    -d '{"url":"https://google.co.jp"}' http://localhost:9515/session/${SESSION_ID}/url

You will be redirected to the top page of Google. The ${SESSION_ID} is the one you obtained earlier.

3. Getting elements & Keyboard input

Now let's do some Selenium-like things. First, we'll do something like findElementByName() to get the element from the name attribute.

curl -X POST -H 'Content-Type: application/json' \
    -d '{"using":"name", "value": "q"}' http://localhost:9515/session/${SESSION_ID}/element \
    | jq -r '.value.ELEMENT'

You can get the element with {"using": "property name", "value": "value"}. Google's search box has name="q", so you can specify it as {"using": "name", "value": "q"}.

You can get the element ID of the search box by executing the above curl.

$ curl ... http://localhost:9515/session/${SESSION_ID}/element | jq -r '.value.ELEMENT'
# Get the element ID
0.5757440182130869-1

An "element ID" is required to perform any action using the element, such as clicking on the element, keyboard input, or focus.

In the case of keyboard input, it looks like this

curl -s -X POST -H 'Content-Type: application/json' \
-d '{"value": ["animal\n"]}' http://localhost:9515/session/${SESSION_ID}/element/${elementId}/value

The only thing you need to be careful about is to pass the string to be input in array format. You can also add \n to the end to make it act as an ENTER.

Works like Selenium

If you combine the above into a single ShellScript and run it, you have a complete Selenium.

#!/usr/bin/env bash

main() {
    # Open the URL
    navigate_to 'https://google.co.jp'

    # Get the search box
    local searchBox=$(find_element 'name' 'q')

    # keyboard input and searching
    send_keys $searchBox "animal\n"
}

get_session_id() {
    curl -s -X POST -H 'Content-Type: application/json' \
        -d '{
            "desiredCapabilities": {
                "browserName": "chrome"
            }
        }' \
        ${ROOT}/session | jq -r '.sessionId'
}

ROOT=http://localhost:9515
SESSION_ID=$(get_session_id)
BASE_URL=${ROOT}/session/${SESSION_ID}


navigate_to() {
    local url=$1
    curl -s -X POST -H 'Content-Type: application/json' -d '{"url":"'${url}'"}' ${BASE_URL}/url
}

find_element() {
    local property=$1
    local value=$2
    curl -s -X POST -H 'Content-Type: application/json' \
    -d '{"using":"'$property'", "value": "'$value'"}' ${BASE_URL}/element | jq -r '.value.ELEMENT'
}

send_keys() {
    local elementId=$1
    local value=$2
    curl -s -X POST -H 'Content-Type: application/json' \
    -d '{"value": ["'$value'"]}' ${BASE_URL}/element/${elementId}/value >/dev/null
}

main

It's very simple: by making the part of curl that throws JSON into a small function, if you just look at the contents of the main function, it's already a complete WebDriver description.
The parts that have been put together into functions can be put together in selenium.sh, etc. and sourced as shown below for a very simple look.

#!/usr/bin/env bash
source ./selenium.sh

main() {
    # Open the URL
    navigate_to 'https://google.co.jp'

    # Get the search box
    local searchBox=$(find_element 'name' 'q')

    # keyboard input and searching
    send_keys $searchBox "animal\n"
}
main

Bonus: iTerm 🤝 Selenium

So, I attached iTerm and Selenium together with ShellScript.

While working with iTerm, the browser is automatically operating behind the scenes.

I've created a repository for shellnium, so have a look at that too!
https://github.com/Rasukarusan/shellnium

Reference

WebDriver

Make Button By Floating Window

Rasukarusan — Sun, 14 Feb 2021 12:12:47 +0000

Let's make a button by floating window!

I made a button using floating window. You can push the button by ENTER.

How do I make it look like a button?

Prepare two windows, front and back.
Make the color of the back a little darker than the color of the front.
Lower the position of the front so that it covers the back when it is pressed.

When ENTER is pressed, just move it back down and it will appear to be pressed.

Code

let g:button_window = {}
hi FrontColor guibg=#F27200
hi BackColor guibg=#AC5D24

function! s:center(str)
  let width = nvim_win_get_width(0)
  let shift = floor(width/2) - floor(strdisplaywidth(a:str)/2)
  return repeat(' ', float2nr(shift)) . a:str
endfunction

function! s:remove_button() abort
  let front_win = nvim_get_current_win()
  let back_win = g:button_window[front_win]
  call nvim_win_close(back_win, v:true)
  call nvim_win_close(front_win, v:true)
  call remove(g:button_window, front_win)
endfunction

function! s:create_window(config, ...) abort
  let hi_group = a:1
  let transparency = get(a:, '2', 0)
  let buf = nvim_create_buf(v:false, v:true)
  let win = nvim_open_win(buf, v:true, a:config)
  if hi_group != ''
    call nvim_win_set_option(win, 'winhighlight', hi_group)
    call nvim_win_set_option(win, 'winblend', transparency)
    call nvim_win_set_config(win, a:config)
  endif
  return win
endfunction

function! s:push() abort
  let front_win = nvim_get_current_win()
  let back_win = g:button_window[front_win]

  let config = nvim_win_get_config(front_win)
  let config.row += 1
  call nvim_win_set_config(front_win, config)

  sleep 100ms
  redraw

  let config.row -= 1
  call nvim_win_set_config(front_win, config)
endfunction

function! s:main() abort
  let row = 20
  let col = 20
  let width = 20
  let height = 3
  let config = { 'relative': 'editor', 'row': row, 'col': col, 'width': width, 'height': height, 'anchor': 'NW', 'style': 'minimal' }

  " back window
  let back_config = deepcopy(config)
  let back_config.row += 1
  let back_win = s:create_window(back_config, 'Normal:BackColor')

  " front window
  let front_win = s:create_window(config, 'Normal:FrontColor')
  call setline(2, s:center('Button'))
  call cursor(2, 0)
  nnoremap <buffer><nowait><silent> :q :call <SID>remove_button()
  nnoremap <CR> :call <SID>push()<CR>

  let g:button_window[front_win] = back_win
endfunction

call s:main()

Bonus: Fire bullets with ENTER

And you can move by hjkl.

let g:button_window = {}
hi FrontColor guibg=#F27200
hi BackColor guibg=#AC5D24

function! s:center(str)
  let width = nvim_win_get_width(0)
  let shift = floor(width/2) - floor(strdisplaywidth(a:str)/2)
  return repeat(' ', float2nr(shift)) . a:str
endfunction

function! s:move(direction, value)
  let front_win = nvim_get_current_win()
  let back_win = g:button_window[front_win]
  for id in [front_win, back_win]
    let config = nvim_win_get_config(id)
    if a:direction == 'x'
      let config.col += a:value
    else
      let config.row += a:value
    endif
    call nvim_win_set_config(id, config)
  endfor
endfunction

function! s:remove_button() abort
  let front_win = nvim_get_current_win()
  let back_win = g:button_window[front_win]
  call nvim_win_close(back_win, v:true)
  call nvim_win_close(front_win, v:true)
  call remove(g:button_window, front_win)
endfunction

function! s:create_window(config, ...) abort
  let hi_group = a:1
  let transparency = get(a:, '2', 0)
  let buf = nvim_create_buf(v:false, v:true)
  let win = nvim_open_win(buf, v:true, a:config)
  if hi_group != ''
    call nvim_win_set_option(win, 'winhighlight', hi_group)
    call nvim_win_set_option(win, 'winblend', transparency)
    call nvim_win_set_config(win, a:config)
  endif
  return win
endfunction

function! s:fire() abort
  let front_win = nvim_get_current_win()
  let conf = nvim_win_get_config(front_win)
  let row = conf.row + 1
  let col = conf.col + conf.width
  let width = 2
  let height = 1
  let config = { 'relative': 'editor', 'row': row, 'col': col, 'width': width, 'height': height, 'anchor': 'NW', 'style': 'minimal', }
  let ballet = s:create_window(config, 'Normal:FrontColor')

  for i in range(1, 30)
    let config.col += 1
    call nvim_win_set_config(ballet, config)
    redraw
    sleep 10ms
  endfor

  call nvim_win_close(ballet, v:true)
endfunction

function! s:push() abort
  let front_win = nvim_get_current_win()
  let back_win = g:button_window[front_win]
  let config = nvim_win_get_config(front_win)
  let config.row += 1
  call nvim_win_set_config(front_win, config)

  sleep 100ms
  redraw

  let config.row -= 1
  call nvim_win_set_config(front_win, config)

  call s:fire()
endfunction

function! s:main() abort
  let row = 20
  let col = 20
  let width = 20
  let height = 3
  let config = { 'relative': 'editor', 'row': row, 'col': col, 'width': width, 'height': height, 'anchor': 'NW', 'style': 'minimal' }

  " back window
  let back_config = deepcopy(config)
  let back_config.row += 1
  let back_win = s:create_window(back_config, 'Normal:BackColor')

  " front window
  let front_win = s:create_window(config, 'Normal:FrontColor')
  call setline(2, s:center('Button'))
  call cursor(2, 0)

  nnoremap <buffer><nowait><silent> l :call <SID>move('x', 2)<CR>
  nnoremap <buffer><nowait><silent> h :call <SID>move('x', -2)<CR>
  nnoremap <buffer><nowait><silent> j :call <SID>move('y', 2)<CR>
  nnoremap <buffer><nowait><silent> k :call <SID>move('y', -2)<CR>
  nnoremap <buffer><nowait><silent> :q :call <SID>remove_button()
  nnoremap <CR> :call <SID>push()<CR>

  let g:button_window[front_win] = back_win
endfunction

call s:main()

Have a happy vim life!