DEV Community: Yusuke Iwaki

Ruby on Rails System testing (E2E testing) using Natural Language with OpenAI API

Yusuke Iwaki — Fri, 20 Dec 2024 20:04:56 +0000

Background

End-to-end (E2E) testing can be challenging to maintain.

For instance, even a simple login screen can have various variations. As a service evolves, there often arises a need to upgrade a straightforward login screen to a more advanced one with enhanced functionality.

As shown in the figure below, what appears to humans as the same "screen for entering ID and password" can be vastly different for machines. It is not easy for a system to identify both interfaces as login screens and test them with the same script.

What Is the Problem?

E2E testing is not a one-and-done task; it needs to adapt to changes in the user interface (UI). Otherwise, the cost of updating the tests to keep up with UI changes can outweigh the benefits of ensuring the functionality specifications through testing.

In current common practices for E2E testing, UI elements such as buttons and text boxes are selected using CSS selectors or ARIA (Accessibility). These elements are then interacted with, such as clicking. However, even for the same login screen, relying solely on mechanical selection via CSS selectors makes it difficult to accommodate UI changes.

Additionally, when humans conduct tests, they attempt to recover from operational errors along the way. On the other hand, mechanical testing lacks such adaptability. For instance, if login information is mistakenly omitted and a submission results in an error, the automated test engine may still try to proceed with operations as if the login were successful. This is a common cause of what is known as a "flaky test."

Writing Ruby on Rails System Tests in Natural Language!

With the remarkable advancements in generative AI, it is now conceivable to devise a method that allows tests to be written in natural language and executed directly. Inspired by this idea, I have developed a prototype and published it as a Rubygem.

https://github.com/YusukeIwaki/charai

before do
  Capybara.current_driver    = :charai
  Capybara.javascript_driver = :charai

  page.driver.additional_instruction = <<~MARKDOWN
  * Our website consists of 2-pain layout. The core function of this web service is helping users to find a good job offer.
  * On the left-pain, we can filter job offers by text search queries and conditions by checkboxes. On the center-pain, job offers are shown. It contains 30 items, and the second item can be a sponsored advertisement item.
  * The center-pain is scrollable while the left-pain is not. So please keep it mind that we have to put our mouse cursor to the center-pain for scrolling.
  MARKDOWN
end

it 'should work' do
  page.driver << <<~MARKDOWN
  * Browse to the job offer list view. Then input "Ruby on Rails".
  * After inputing the search query, submit it and check if the job offers shown in the center-pain is really related to Ruby on Rails.
  * If most of the search result is irrelevant to Ruby on Rails, please mark this testcase as "Assertion failure".
  MARKDOWN
end

Directions

Traditionally, natural language commands are converted into CSS selectors for execution. While this approach can adapt UI changes, it does not resolve the issue of recovering from operational errors.

To overcome this limitation, this library employs a unique approach: the AI autonomously observes the screen and generates Ruby code for performing the required actions. The generated Ruby code is executed, and the resulting screenshots are feedback into the AI for analysis. Based on this analysis, the next steps are determined and executed. This process essentially models how humans perform manual testing.

By adopting this method, even if an click opens an unintended page, the AI can analyze the screenshot of the unexpected page, issue instructions to return to the previous page, and continue testing from there. This approach closely mimics the way humans recover from errors during testing, ensuring robustness and adaptability.

How to implement it?

To implement this approach in Ruby on Rails, creating a custom Capybara driver is the simplest method.

There is plenty of information available via a quick Google search on how to use a Capybara driver. However, guidance on creating a Capybara driver is scarce. You would need to study the Capybara source code or refer to the implementation of existing libraries like Cuprite, and then replicate their techniques to build your own driver.

Traditional Capybara drivers offer a wide range of methods under the Capybara DSL, such as click, find, and set. However, the driver we aim to create simplifies this dramatically by providing only three methods:

visit Method: This is used to navigate to the initial web page.
Precondition Method: This method communicates the preconditions for the test.
Test Content Method: This method conveys the actual test instructions to be executed.

By limiting the functionality to these three essential methods, this new driver focuses on allowing AI-driven natural language instructions to handle dynamic testing processes. This approach aligns closely with the goal of creating adaptive and resilient E2E tests.

More detailed instruction to implement the driver is introduced in this movie and deck (Sorry in Japanese!!)

Does it really work??

At this moment, it works... to some extent. The current accuracy is still low, and it is not yet production-ready. However, this approach holds the potential to significantly transform the development experience of writing and running system tests in Rails.

The prototype showcases how leveraging AI in this manner could revolutionize E2E testing by making it more adaptable and human-like. While there is much room for improvement, the foundation is promising, and further refinements could lead to practical, impactful solutions in the future.

So please try it, and give me your feedback! :)

YusukeIwaki / charai

Charai(Chat + Ruby + AI) driver for Capybara. Prototype impl for Kaigi on Rails 2024 presentation.

Charai

Chat + Ruby + AI = Charai

Setup

Add gem 'charai' into your project's Gemfile, and then bundle install

Also, this gem requires Firefox Developer Edition to be installed on the location below:

/Applications/Firefox Developer Edition.app (macOS)
/usr/bin/firefox-devedition (Linux)

Configuration

Configure your Capybara driver like below.

Capybara.register_driver :charai do |app|
  Charai::Driver.new(app, openai_configuration: config)
end

Capybara.register_driver :charai_headless do |app|
  Charai::Driver.new(app, openai_configuration: config, headless: true)
end

Please note that this driver required OpenAI service.

OpenAI

config = Charai::OpenaiConfiguration.new(
  model: 'gpt-4o',
  api_key: 'sk-xxxxxxxxxxx'
)

Azure OpenAI (Recommended)

config = Charai::AzureOpenaiConfiguration.new(
  endpoint_url: 'https://YOUR-APP.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-05-01-preview',
  api_key: 'aabbcc00112233445566'
)

Usage

Since this driver works with the OpenAI service, we can easily describe E2E test like below :)

before do
  Capybara.current_driver    = :charai
  Capybara.javascript_driver = :charai
  page.driver.additional_instruction = <<~MARKDOWN
  * このページは、3ペイン構造です。ユーザが仕事を探すためのページです。
  * 左ペインには仕事の絞り込みができるフィルター、中央ペインが仕事（求人）の一覧で、30件ずつ表示されます。
  * 左ペインにマウスを置いてスクロールしても、中央ペインはスクロールされません。一覧をスクロールしたいときには、中央ペインの座標を確認し、その中央にマウスを置いてスクロールしてください。
  * 右ペインは、広告エリアです。検索条件に応じた広告が表示されます。
  MARKDOWN
end

it 'should work' do
  page.driver << <<~MARKDOWN
  *

…

View on GitHub

Backport `Data.define` for Ruby 2.7, 3.0, 3.1

Yusuke Iwaki — Wed, 08 Mar 2023 04:17:00 +0000

Data.define is introduced by Zverok in Ruby 3.2.

New in Ruby 3.2 - Data.define

Brandon Weaver ・ Oct 6 '22 ・ 2 min read

This is really amazing and desired by most Rubyists. However, regardless to say, the function is not available in Ruby 2.7-3.1 applications. Some users (including me), at this moment, still uses older versions of Ruby with struggling to upgrade large Rails applications.

data_class_factory

I developed a backport of Data.define for Ruby 2.7, 3.0, 3.1.
https://rubygems.org/gems/data_class_factory

require 'data_class_factory'

Point = Data.define(:x, :y) do
  def norm
    Math.sqrt(x * x + y * y)
  end
end

p1 = Point.new(x: 3, y: 4)
p2 = Point.new(x: 3, y: 4)

p1 == p2 # => true
p1.norm # => 5

It covers most of the spec of original Data class introduced in Ruby 3.2.
https://github.com/YusukeIwaki/data_class_factory/blob/0.1.0/test/data_test.rb

Feedback is really welcome :)

I personally use this library on my Rails apps. So try this lib for your app, and let me know issues and feel free to feedback.
https://github.com/YusukeIwaki/data_class_factory/issues

Dart + GitHub Actions: Publish your command-line tools for Windows/macOS/Linux into GitHub Releases

Yusuke Iwaki — Sat, 02 Apr 2022 06:29:10 +0000

Motivation

Dart is very suitable for building quick-and-dirty CLI tools and share them among your collegue, since it produces a single binary with dart compile exe.

It would be useful if organization-internal tools are published in GitHub Releases like this:

Unfortunately, Dart doesn't support cross-compile at this moment and we have to compile the source code on each platform (Windows, macOS, Linux) before publishing the binaries, while Golang supports cross-compile and GoReleaser already made it easy to publish the binaries for each platform.

GitHub Actions provides Windows/macOS/Linux build environments for free and it would be really useful for compiling and publishing the CLI tools composed with Dart.

This article shows how to configure the GitHub Actions.

GitHub Actions for build

As many articles already says, only 4 step are required for compiling the Dart code.

- uses: actions/checkout@v2
- uses: dart-lang/setup-dart@v1
- run: dart pub get
- run: dart compile exe bin/mycli.dart -o mycli

One point to consider is that the output binary should be named as mycli .exe on Windows and mycli (without extension) on macOS/Linux.
We can solve the naming by defining the matrix like below:

jobs:
  compile:
    name: dart compile exe
    strategy:
      matrix:
        include:
          - runs-on: ubuntu-latest
            binary-name: mycli_linux_amd64
          - runs-on: macos-latest
            binary-name: mycli_macos_amd64
          - runs-on: windows-latest
            binary-name: mycli_windows.exe
    runs-on: ${{ matrix.runs-on }}
    steps:
      - uses: actions/checkout@v2
      - uses: dart-lang/setup-dart@v1
      - run: dart pub get
      - run: mkdir ${{ matrix.runs-on }}
      - run: dart compile exe bin/mycli.dart -o ${{ matrix.runs-on }}/${{ matrix.binary-name }}

GitHub Actions for release

https://docs.github.com/en/rest/reference/releases

According to this reference, GitHub Release requires 2 steps for publishing some binaries:

Create a release resource
- POST https://api.github.com/repos/{owner}/{repo}/releases
Upload the binaries into the release
- POST https://uploads.github.com/repos/{owner}/{repo}/releases/{release_id}/assets

Use action-gh-release for simple uploading

action-gh-release really makes it easy to do these steps.

- uses: softprops/action-gh-release@v1
  with:
    draft: true
    files: out/*

With this simple definition, GitHub Actions will upload all files in out/ into a new draft release.

    files: |
      out/mycli_linux
      out/mycli_macos
      out/mycli_windows

Also files can be specified to individual file list instead of glob expression.

Combine the build process with the release process using artifact

Each of 3 build actions generate a binary and release process should upload them all at once. The flow can be implemented with GitHub artifact.

name: Publish

on:
  push:
    tags:
      - '[0-9]+.[0-9]+.[0-9]+'

jobs:
  compile:
    name: dart compile exe
    strategy:
      matrix:
        include:
          - runs-on: ubuntu-latest
            binary-name: mycli_linux_amd64
          - runs-on: macos-latest
            binary-name: mycli_macos_amd64
          - runs-on: windows-latest
            binary-name: mycli_windows.exe
    runs-on: ${{ matrix.runs-on }}
    steps:
      - uses: actions/checkout@v2
      - uses: dart-lang/setup-dart@v1
      - run: dart pub get
      - run: mkdir ${{ matrix.runs-on }}
      - run: dart compile exe bin/mycli.dart -o ${{ matrix.runs-on }}/${{ matrix.binary-name }}
      - uses: actions/upload-artifact@v2
        with:
          name: bin-${{ matrix.runs-on }}
          path: ${{ matrix.runs-on }}

  release:
    needs: compile
    name: github release
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v2
        with:
          name: bin-ubuntu-latest
          path: bin-linux
      - uses: actions/download-artifact@v2
        with:
          name: bin-macos-latest
          path: bin-macos
      - uses: actions/download-artifact@v2
        with:
          name: bin-windows-latest
          path: bin-windows
      - uses: softprops/action-gh-release@v1
        with:
          draft: true
          files: bin-*/*

This works expectedly like below:

and the artifacts are successfully uploaded to GitHub Releases! :)

System testing for Rails application without Capybara, using Puppeteer

Yusuke Iwaki — Fri, 03 Sep 2021 05:58:03 +0000

Refer this article instead if you can read Japanese sentences.
日本語を読める方はこちらの記事を見てください

Background

Rails introduced "system testing", that makes it easy to configure Capybara: acceptance test framework for Ruby. We may often use it not for "acceptance test" but for just UI testing of the Rails application with React/Vue or other rich JavaScript-powered features.

Capybara DSL is actually a little inaccurate for working with pages with rich JavaScript features.

Capybara has internally polling with interval 10msec for waiting for Elements available and actionable.
https://github.com/teamcapybara/capybara/blob/0468de5a810aae75ab9de20447e246c5c35473f0/lib/capybara/node/base.rb#L91
I don't know this is really the main cause, but we may often face flaky/unstable testcases in using Capybara.

On the other hand, Puppeteer or Playwright have a relyable solution for waiting for Elements: Page#waitForSelector and Page#waitForNavigation. (Playwright has more advanced feature of auto-waiting )

I actually ported the libraries for using them in Rails application.

Playwright for Ruby: https://playwright-ruby-client.vercel.app/
Puppeteer for Ruby: https://github.com/YusukeIwaki/puppeteer-ruby

However the relyable features cannot be used from Capybara DSL. So let's try to configure system tests without Capybara, and using Puppeteer.

What happens if Capybara is absent??

Since Capybara is a huge framework, it would be difficult for some users to imagine what happens without Capybara.

Actually Rails application without Capybara loses the functionalities

Launching test HTTP server on starting tests
Capybara::DSL
- Using visit or other methods raises NoMethodError instead of NotImplementedError
- System spec (on RSpec) or SystemTestCase (on MiniTest), which depends on Capybara::DSL

However rspec-rails still provides features like

Attaching type: :feature type: :system to example.metadata
Defining feature background scenario methods for feature spec

So actually all we have to prepare is

Launching test HTTP server
Performing automation with browsers
- This will be satisfied by Puppeteer

Launching test HTTP server without Capybara

Capybara is really kind that

finds available port
allows to configure host
allows to launch Webrick and Puma

However let's forget them at this moment, and assume that we only have to launch Puma with the baseURL http://127.0.0.1:3000

With referring the logics of Capybara preparing server and launching Puma server, then we can easily find the minimum configuration for launching test server like below:

RSpec.configure do |config|
  config.before(:suite) do
    # launching Rails server for system testing
    require 'rack/builder'
    testapp = Rack::Builder.app(Rails.application) do
      map '/__ping' do # debugging endpoint for heartbeat
        run ->(env) { [200, { 'Content-Type' => 'text/plain' }, ['OK']] }
      end
    end

    require 'rack/handler/puma'
    server_thread = Thread.new do
      Rack::Handler::Puma.run(testapp,
        Host: '127.0.0.1',
        Port: 3000,
        Threads: '0:4',
        workers: 0,
        daemonize: false,
      )
    end

    # Waiting for Rails server is ready, using Net::HTTP
    require 'net/http'
    require 'timeout'
    Timeout.timeout(3) do
      loop do
        puts Net::HTTP.get(URI("http://127.0.0.1:3000/__ping"))
        break
      rescue Errno::EADDRNOTAVAIL
        sleep 1
      rescue Errno::ECONNREFUSED
        sleep 0.1
      end
    end
  end

  # Configure puppeteer-ruby for automation
  config.around(type: :feature) do |example|
    Puppeteer.launch(channel: :chrome, headless: false) do |browser|
      @server_base_url = 'http://127.0.0.1:3000'
      @puppeteer_page = browser.new_page
      example.run
    end
  end
end

By putting this configuration into spec/support/integration_test_helper.rb or another place you prefer, now we can enjoy system testing with Puppeteer like below!!

As is already described, we cannot use system spec. Use feature spec (provided by rspec-rails) instead.

require 'rails_helper'

describe 'example' do
  let(:base_url) { @server_base_url }
  let(:page) { @puppeteer_page }

  let(:user) { FactoryBot.create(:user) }

  it 'can browse' do
    page.goto("#{base_url}/tests/#{user.id}")
    page.wait_for_selector('input', visible: true)
    page.type_text('input', 'hoge')
    page.keyboard.press('Enter')

    text = page.eval_on_selector('#content', 'el => el.textContent')
    expect(text).to include('hoge')
    expect(text).to include(user.name)
  end
end

A little refactoring for production use

The logic of launching test server, introduced in the previous section, is really straightforward. However most users would hesitate to copy/paste it into your own products because it's too dirty :(

No worry, here is a better code with more relyable Rack::Server.start to launch HTTP server, which is actually used in the implementation of rackup command

class RackTestServer
  def initialize(app:, **options)
    @options = options

    @options[:Host] ||= '127.0.0.1'
    @options[:Port] ||= 3000

    require 'rack/builder'
    @options[:app] = Rack::Builder.app(app) do
      map '/__ping' do
        run ->(env) { [200, { 'Content-Type' => 'text/plain' }, ['OK']] }
      end
    end
  end

  def base_url
    "http://#{@options[:Host]}:#{@options[:Port]}"
  end

  def start
    require 'rack/server'
    Rack::Server.start(**@options)
  end

  def ready?
    require 'net/http'
    begin
      Net::HTTP.get(URI("#{base_url}/__ping"))
      true
    rescue Errno::EADDRNOTAVAIL
      false
    rescue Errno::ECONNREFUSED
      false
    end
  end

  def wait_for_ready(timeout: 3)
    require 'timeout'
    Timeout.timeout(3) do
      sleep 0.1 until ready?
    end
  end
end

With this test server, the RSpec configuration file can be much shorter and simplified :)

RSpec.configure do |config|
  config.before(:suite) do
    # Launch Rails application
    test_server = RackTestServer.new(
      # options for Rack::Server
      # https://github.com/rack/rack/blob/2.2.3/lib/rack/server.rb#L173
      app: Rails.application,
      server: :puma,
      Host: '127.0.0.1',
      Port: 3000,
      daemonize: false,

      # options for Rack::Handler::Puma
      # https://github.com/puma/puma/blob/v5.4.0/lib/rack/handler/puma.rb#L84
      Threads: '0:4',
      workers: 0,
    )

    Thread.new { test_server.start }
    test_server.wait_for_ready
  end

  # Configure puppeteer-ruby for automation
  config.around(type: :feature) do |example|
    Puppeteer.launch(channel: :chrome, headless: false) do |browser|
      @server_base_url = 'http://127.0.0.1:3000'
      @puppeteer_page = browser.new_page
      example.run
    end
  end
end

Now, you can now get rid of Capybara from your app if you just want to launch HTTP server for testing :)

You can also use rack-test_server Gem instead of defining your own RackTestServer.

Using 🎭Playwright in Ruby/Rails

Yusuke Iwaki — Tue, 10 Aug 2021 14:25:06 +0000

🎭Playwright is a really awesome library for browser automation. Microsoft already published several language bindings.

TypeScript: https://github.com/microsoft/playwright
Java (or Kotlin): https://github.com/microsoft/playwright-java
Python: https://github.com/microsoft/playwright-python
C#: https://github.com/microsoft/playwright-dotnet

And also playwright-go is planned to be official.
https://github.com/microsoft/playwright/issues/6856

No Ruby client?

Ruby or Rails developers would like to use Playwright for acceptance testing or other use-cases, but it is not available at this moment.

Playwright team explicitly says there is no plan to release Ruby client.
https://github.com/microsoft/playwright/issues/5874

Ruby users should be happy with Playwright!!

It is clear that Ruby is not so famous as Python or TypeScript, but many Rails users still use it.

Rails already included an acceptance testing framework (called SystemTestCase). It uses Capybara and most users already noticed Capybara tends to produce flaky testcases.

Playwright provides very clever DOM selectors with auto-waiting feature, and useful utility functions. As Internet Explorer is already dead, no reason to keep using Selenium and Capybara.

Fortunately, Playwright is server-client architecture. We can use the useful features on any languages by just implementing a Playwright API client.

Try it on Ruby!!

I actually created a Playwright client for Ruby.
https://github.com/YusukeIwaki/playwright-ruby-client



Playwright.connect_to_playwright_server('ws://127.0.0.1:8080') do |playwright|
  playwright.chromium.launch do |browser|
    page = browser.new_page
    page.goto('https://github.com/YusukeIwaki')
    page.screenshot(path: './YusukeIwaki.png')
  end
end

Note that we have to launch Playwright server in advance, like this: npx playwright install && npx playwright run-server 8080

Try it on Rails!!

I also created a Capybara driver of Playwright.
https://github.com/YusukeIwaki/capybara-playwright-driver



Capybara.register_driver(:playwright) do |app|
  Capybara::Playwright::Driver.new(app, browser_type: :firefox, headless: false)
end
Capybara.default_max_wait_time = 15
Capybara.default_driver = :playwright

Capybara DSL is supported, and we can also use Playwright-native features such as recording video during testing.
https://playwright-ruby-client.vercel.app/docs/article/guides/rails_integration#available-functions-and-limitations

Conclusion

Playwright's architecture is great, and we Ruby/Rails users can use Playwright features by developing our own Playwright client API library.

Unfortunately Microsoft doesn't make effort for Ruby users, and difficult to ask for official support for Ruby.

Using 🎭Playwright in python:alpine

Yusuke Iwaki — Tue, 10 Aug 2021 13:41:26 +0000

playwright-python is a really awesome library for browser automation.

However it doesn't support running on Alpine Linux.
https://github.com/microsoft/playwright-python/issues/234

Most web developers would use Alpine based Docker images, and actually playwright-python internally has a capability of running scripts in Alpine.

Why playwright-python cannot run on Alpine?

playwright-python comes with a browser downloader. As we know, browsers requires many dependencies. The deps are not compatible with Alpine.

This might be a main reason why playwright-python deny running on Alpine.

By the way, we already know Selenium Remote driver (Selenium Grid), it separates script runner and browser environments communicating with HTTP port (default: 4444).

How to separate script runner and browsers

playwright-python is just a Python client library of Playwright. Playwright server communicates with browser via WebSocket or other protocols, and playwright-python communicates with the server via pipe by default.

However playwright-python already introduced WebSocketTransport for the API BrowserType#connect() in Playwright v1.11 and later. It can be also used for Playwright connection between client and server.

Try it!

I created a tiny library.
https://github.com/YusukeIwaki/playwright-python-remote

This library provides a Playwright initializer (ContextManager) just replacing PipeTransport with WebSocketTransport.
https://github.com/YusukeIwaki/playwright-python-remote/blob/main/playwright_remote/sync_api/sync_playwright_remote.py

Playwright Server can be prepared very easily with Docker like this:



FROM mcr.microsoft.com/playwright:focal

WORKDIR /root
RUN npm install playwright@1.12.3 && ./node_modules/.bin/playwright install
CMD ["./node_modules/.bin/playwright", "run-server"]

Now we can enjoy playwright-python on Alpine:



from playwright_remote.sync_api import sync_playwright_remote

with sync_playwright_remote('ws://127.0.0.1:8080/ws') as playwright:
  browser = playwright.chromium.launch()
  try:
    page = browser.new_page()
    page.goto('https://github.com/YusukeIwaki')
    page.screenshot(path='YusukeIwaki.png')
  finally:
    browser.close()

I also tried it on Heroku: https://zenn.dev/yusukeiwaki/scraps/3ad5f8ed536720 (Sorry, Japanese only...)

Conclusion

playwright-python is really awesome, it runs on Alpine actually.

Note that WebSocketTransport (introduced in this article) is internal class and can be changed in future releases.

Just expect the feature is officially supported. Let's upvote
https://github.com/microsoft/playwright/issues/4687 👍