DEV Community: Yusuke Iwaki

Introducing pytest-style fixtures into Ruby for smarter browser testing

Yusuke Iwaki — Sun, 03 May 2026 16:22:00 +0000

Why I built another Ruby test runner inspired by Playwright Test

Ruby already has great testing tools.

If you are building Rails applications today, you probably use one of these combinations:

RSpec + Capybara
Minitest + Capybara
Rails system tests
Maybe Selenium, Cuprite, Ferrum, or Playwright through Ruby bindings

These tools are mature, battle-tested, and widely used.

So the natural question is:

Why create yet another test runner?

That is exactly the question the Playwright team asked themselves when they created Playwright Test.

In the Playwright video that inspired this project, the speaker says they did not originally want to create another JavaScript testing framework. They tried existing frameworks such as Mocha and Jest, but those frameworks had historically been designed for unit testing. End-to-end testing had a different set of problems: cross-browser execution, parallelization, browser-side isolation, and a high degree of fixture flexibility.

That explanation clicked with me.

Because Ruby on Rails browser testing has a similar problem.

RSpec and Minitest are excellent general-purpose Ruby test frameworks. Capybara is an excellent browser automation DSL. But the current mainstream Rails browser testing style was not designed after learning from Playwright Test. It still feels like a unit-test-era architecture with browser automation added on top.

Playwright Test showed that browser testing deserves a test runner designed around browser testing itself.

That is the idea behind Smartest.

Repository:

YusukeIwaki / smartest

Introduce Pytest-style fixtures for Ruby. A tiny test runner with keyword-argument fixture injection, dependencies, and cleanup.

Smartest

Smartest is a Ruby test runner that brings pytest-style fixtures to Ruby with explicit fixture dependencies, automatic teardown, and Playwright-friendly browser testing.

Tests request fixtures with Ruby keyword arguments. Fixtures define their own dependencies the same way:

class WebFixture < Smartest::Fixture
  fixture :server do
    server = TestServer.start
    on_teardown { server.stop }
    server
  end

  fixture :client do |server:|
    Client.new(base_url: server.url)
  end
end

around_suite do |suite|
  use_fixture WebFixture
  suite.run
end

test("GET /health") do |client:|
  response = client.get("/health")

  expect(response.status).to eq(200)
end

Smartest is designed around three ideas:

Tests should be readable at the top level.
Fixture dependencies should be explicit.
Teardown should be written only when it is needed.

Why Smartest?

Ruby…

View on GitHub

Documentation: https://smartest-rb.vercel.app/
Browser testing guide: https://smartest-rb.vercel.app/docs/playwright-browser-tests

The inspiration: Playwright Test was not just another runner

Playwright Test is not only a wrapper around browser automation.

It is a test runner designed for the real constraints of end-to-end testing:

cross-browser support out of the box
parallel execution
browser-context isolation
web-first assertions
traceability and debugging support
fixtures inspired by pytest

The official Playwright repository describes Playwright Test as a full-featured test runner for end-to-end testing, with Chromium, Firefox, and WebKit support, full browser isolation, auto-waiting, and web-first assertions.

The important point is not simply “Playwright controls browsers.”

The important point is:

Playwright Test changed the test runner because the target domain changed.

Browser tests are not just unit tests with a browser.

They need their own setup model.

The fixture idea I wanted in Ruby

One of the best ideas in pytest is fixture injection.

In pytest, a test can request what it needs by naming it:

def test_get_me(logged_in_client):
    response = logged_in_client.get("/me")
    assert response.status_code == 200

The test body does not say how to create logged_in_client.

The fixture system knows how to resolve it.

Fixtures can depend on other fixtures:

@pytest.fixture
def logged_in_client(client, user):
    client.login(user)
    return client

This is extremely useful for browser tests, because browser tests often have layered setup:

app server
browser runtime
browser process
browser context
page
user
login state
seeded data
teardown

RSpec and Minitest can express all of this, of course.

But the dependency graph is often implicit.

With RSpec, we may write something like this:

let(:server) { start_server }
let(:client) { Client.new(base_url: server.url) }
let(:user) { create_user }
let(:logged_in_client) do
  client.login(user)
  client
end

it "GET /me" do
  response = logged_in_client.get("/me")
  expect(response.status).to eq(200)
end

This works.

But the fixture dependency graph is hidden inside method calls.

I wanted this instead:

test("GET /me") do |logged_in_client:|
  response = logged_in_client.get("/me")

  expect(response.status).to eq(200)
end

And fixture definitions like this:

class AppFixture < Smartest::Fixture
  fixture :client do |server:|
    Client.new(base_url: server.url)
  end

  fixture :logged_in_client do |client:, user:|
    client.login(user)
    client
  end
end

The important design decision is the keyword argument.

test("GET /me") do |logged_in_client:|

This makes the test’s dependencies explicit.

And:

fixture :logged_in_client do |client:, user:|

This makes the fixture’s dependencies explicit.

No positional argument order.
No hidden let dependency.
No separate with: [:server] declaration.

The dependency declaration and usage live in one place: the Ruby method signature.

Smartest: a small keyword-fixture-first Ruby test runner

Smartest is a small Ruby test runner with a keyword-fixture-first design.

The goal is not to replace all Ruby testing immediately.

The goal is to explore what Ruby testing could feel like if we applied the lessons of pytest fixtures and Playwright Test to Ruby browser testing.

A normal Smartest test looks like this:

test("factorial") do
  expect(1 * 2 * 3).to eq(6)
end

A fixture-driven test looks like this:

test("GET /me") do |logged_in_client:|
  response = logged_in_client.get("/me")

  expect(response.status).to eq(200)
end

Smartest is designed around three ideas:

Tests should be readable at the top level.
Fixture dependencies should be explicit.
Teardown should be written only when it is needed.

That third point is especially important for browser tests.

Some fixtures are just values.
Some fixtures need teardown.
Some fixtures should live for one test.
Some fixtures should live for the whole suite.

Smartest supports both regular fixtures and suite fixtures:

class PlaywrightFixture < Smartest::Fixture
  suite_fixture :playwright do
    runtime = Playwright.create
    on_teardown { runtime.stop }
    runtime
  end

  suite_fixture :browser do |playwright:|
    browser = playwright.chromium.launch
    on_teardown { browser.close }
    browser
  end

  fixture :page do |browser:|
    context = browser.new_context
    page = context.new_page

    on_teardown { context.close }

    page
  end
end

This expresses the browser lifecycle directly:

start Playwright once
launch a browser once
create a fresh browser context and page per test
close the context after each test
close the browser after the suite
stop Playwright after the suite

That is the kind of lifecycle that is awkward when browser testing is treated as a thin layer on top of a unit testing framework.

Why this matters for Rails browser testing

Rails browser testing has traditionally been shaped by RSpec/Minitest + Capybara.

That combination is powerful, but it also reflects an older mental model:

test framework first
browser automation second
browser lifecycle hidden in configuration
setup split across before, let, support files, shared contexts, and driver config

Playwright Test takes a different position:

browser testing is the domain
isolation is part of the runner
fixture composition is part of the runner
cross-browser execution is part of the runner

Smartest is an attempt to bring that kind of thinking to Ruby.

Not by copying Playwright Test’s TypeScript API exactly.

But by asking:

What would a Ruby-native pytest-style fixture system for browser testing look like?

The current answer is:

test("finds the smartest gem on RubyGems") do |page:|
  page.goto("https://rubygems.org/")
  page.locator("input[name='query']").fill("smartest")
  page.keyboard.press("Enter")
  page.locator("a[href='/gems/smartest']").click

  expect(page).to have_url("https://rubygems.org/gems/smartest")
  expect(page.locator("h1")).to have_text("smartest")
end

That page: fixture is the key.

It looks small.

But behind it, Smartest can manage the Playwright runtime, browser process, browser context, page creation, and teardown.

Getting started with browser tests

Smartest includes a browser-test scaffold.

Add Smartest to your project:

gem "smartest"

Then run:

bundle install
bundle exec smartest --init-browser

The browser scaffold sets up the files needed to run Playwright browser tests from Ruby.

It creates files such as:

smartest/test_helper.rb
smartest/fixtures/playwright_fixture.rb
smartest/matchers/playwright_matcher.rb
smartest/example_browser_test.rb

It also installs the browser testing dependencies:

playwright-ruby-client
the Playwright Node.js package
browser binaries

Then run the generated example:

bundle exec smartest smartest/example_browser_test.rb

You can also choose a browser with environment variables:

BROWSER=firefox bundle exec smartest smartest/example_browser_test.rb

Or watch the browser while the test runs:

HEADLESS=false SLOW_MO=250 bundle exec smartest smartest/example_browser_test.rb

The generated fixture uses this lifecycle:

suite_fixture :playwright
  -> starts Playwright runtime once

suite_fixture :browser
  -> launches one browser process for the suite

fixture :page
  -> creates a fresh browser context and page per test

This is intentionally close to the Playwright Test mental model.

AI changed the feasibility of this project

There is another reason I built this now.

A few years ago, building a new test runner would have felt expensive.

You would need to design the DSL, implement the runner, handle fixture resolution, write CLI behavior, create matchers, generate scaffolding, document the behavior, and iterate for a long time.

But with OpenAI Codex, the first working version was surprisingly quick to build.

That changes the economics of experimentation.

I do not mean that AI magically makes design decisions for us.

The important design still comes from human taste:

Should fixtures be positional or keyword-based?
Should fixture dependencies be declared separately or via block parameters?
Should teardown be expressed with yield, ensure, or on_teardown?
Should browser support be built in or generated?
Should this coexist with Minitest and RSpec?

But once the design direction is clear, AI makes it much easier to produce a working prototype and iterate.

In this case, the ideal implementation became possible much faster than I expected.

That matters for open source.

Because many useful developer tools do not start as huge polished projects.

They start as a small experiment with a strong point of view.

This is not “RSpec is bad”

I want to be clear.

RSpec is not bad.
Minitest is not bad.
Capybara is not bad.

They are important parts of the Ruby ecosystem.

But Playwright Test showed something valuable:

Sometimes a new testing domain deserves a new test runner.

Modern browser testing is one of those domains.

If JavaScript testing needed Playwright Test even though Mocha and Jest already existed, then Ruby can also explore a browser-testing-oriented runner even though RSpec and Minitest already exist.

The question is not:

Does Ruby already have testing frameworks?

Of course it does.

The better question is:

Does Ruby have a test runner designed around pytest-style fixture composition and Playwright-style browser lifecycle management?

That is the space Smartest explores.

Current status

Smartest is still young.

It is best understood as a design-stage test runner and a working experiment.

The current focus is:

top-level test
keyword-argument fixture injection
fixture dependencies through keyword arguments
fixture teardown
suite fixtures
around-suite and around-test hooks
browser scaffold generation
Playwright fixture and matcher examples

It is already enough to write Playwright-style browser tests in Ruby.

But it is not yet a mature replacement for RSpec or Minitest.

That is fine.

The goal right now is to validate the idea:

pytest-style fixtures make Ruby browser tests smarter.

Let's Try it!

Install:

gem "smartest"

Initialize browser tests:

bundle exec smartest --init-browser

Run:

bundle exec smartest smartest/example_browser_test.rb

Write:

test("my browser test") do |page:|
  page.goto("https://example.com")

  expect(page.locator("h1")).to have_text("Example Domain")
end

Closing thought

Playwright Test was created because existing JavaScript test frameworks were not built for the unique challenges of end-to-end browser testing.

That lesson should not stay only in the JavaScript world.

Ruby browser testing can also learn from it.

Smartest is my attempt to bring pytest-style fixtures and Playwright-style browser testing ergonomics into Ruby.

Smarter browser testing starts with smarter setup.

Ruby on Rails System testing (E2E testing) using Natural Language with OpenAI API

Yusuke Iwaki — Fri, 20 Dec 2024 20:04:56 +0000

Background

End-to-end (E2E) testing can be challenging to maintain.

For instance, even a simple login screen can have various variations. As a service evolves, there often arises a need to upgrade a straightforward login screen to a more advanced one with enhanced functionality.

As shown in the figure below, what appears to humans as the same "screen for entering ID and password" can be vastly different for machines. It is not easy for a system to identify both interfaces as login screens and test them with the same script.

What Is the Problem?

E2E testing is not a one-and-done task; it needs to adapt to changes in the user interface (UI). Otherwise, the cost of updating the tests to keep up with UI changes can outweigh the benefits of ensuring the functionality specifications through testing.

In current common practices for E2E testing, UI elements such as buttons and text boxes are selected using CSS selectors or ARIA (Accessibility). These elements are then interacted with, such as clicking. However, even for the same login screen, relying solely on mechanical selection via CSS selectors makes it difficult to accommodate UI changes.

Additionally, when humans conduct tests, they attempt to recover from operational errors along the way. On the other hand, mechanical testing lacks such adaptability. For instance, if login information is mistakenly omitted and a submission results in an error, the automated test engine may still try to proceed with operations as if the login were successful. This is a common cause of what is known as a "flaky test."

Writing Ruby on Rails System Tests in Natural Language!

With the remarkable advancements in generative AI, it is now conceivable to devise a method that allows tests to be written in natural language and executed directly. Inspired by this idea, I have developed a prototype and published it as a Rubygem.

https://github.com/YusukeIwaki/charai

before do
  Capybara.current_driver    = :charai
  Capybara.javascript_driver = :charai

  page.driver.additional_instruction = <<~MARKDOWN
  * Our website consists of 2-pain layout. The core function of this web service is helping users to find a good job offer.
  * On the left-pain, we can filter job offers by text search queries and conditions by checkboxes. On the center-pain, job offers are shown. It contains 30 items, and the second item can be a sponsored advertisement item.
  * The center-pain is scrollable while the left-pain is not. So please keep it mind that we have to put our mouse cursor to the center-pain for scrolling.
  MARKDOWN
end

it 'should work' do
  page.driver << <<~MARKDOWN
  * Browse to the job offer list view. Then input "Ruby on Rails".
  * After inputing the search query, submit it and check if the job offers shown in the center-pain is really related to Ruby on Rails.
  * If most of the search result is irrelevant to Ruby on Rails, please mark this testcase as "Assertion failure".
  MARKDOWN
end

Directions

Traditionally, natural language commands are converted into CSS selectors for execution. While this approach can adapt UI changes, it does not resolve the issue of recovering from operational errors.

To overcome this limitation, this library employs a unique approach: the AI autonomously observes the screen and generates Ruby code for performing the required actions. The generated Ruby code is executed, and the resulting screenshots are feedback into the AI for analysis. Based on this analysis, the next steps are determined and executed. This process essentially models how humans perform manual testing.

By adopting this method, even if an click opens an unintended page, the AI can analyze the screenshot of the unexpected page, issue instructions to return to the previous page, and continue testing from there. This approach closely mimics the way humans recover from errors during testing, ensuring robustness and adaptability.

How to implement it?

To implement this approach in Ruby on Rails, creating a custom Capybara driver is the simplest method.

There is plenty of information available via a quick Google search on how to use a Capybara driver. However, guidance on creating a Capybara driver is scarce. You would need to study the Capybara source code or refer to the implementation of existing libraries like Cuprite, and then replicate their techniques to build your own driver.

Traditional Capybara drivers offer a wide range of methods under the Capybara DSL, such as click, find, and set. However, the driver we aim to create simplifies this dramatically by providing only three methods:

visit Method: This is used to navigate to the initial web page.
Precondition Method: This method communicates the preconditions for the test.
Test Content Method: This method conveys the actual test instructions to be executed.

By limiting the functionality to these three essential methods, this new driver focuses on allowing AI-driven natural language instructions to handle dynamic testing processes. This approach aligns closely with the goal of creating adaptive and resilient E2E tests.

More detailed instruction to implement the driver is introduced in this movie and deck (Sorry in Japanese!!)

Does it really work??

At this moment, it works... to some extent. The current accuracy is still low, and it is not yet production-ready. However, this approach holds the potential to significantly transform the development experience of writing and running system tests in Rails.

The prototype showcases how leveraging AI in this manner could revolutionize E2E testing by making it more adaptable and human-like. While there is much room for improvement, the foundation is promising, and further refinements could lead to practical, impactful solutions in the future.

So please try it, and give me your feedback! :)

YusukeIwaki / charai

Charai(Chat + Ruby + AI) driver for Capybara. Prototype impl for Kaigi on Rails 2024 presentation.

Charai

Chat + Ruby + AI = Charai

Setup

Add gem 'charai' into your project's Gemfile, and then bundle install

Also, this gem requires Firefox Developer Edition to be installed on the location below:

/Applications/Firefox Developer Edition.app (macOS)
/usr/bin/firefox-devedition (Linux)

Configuration

Configure your Capybara driver like below.

Capybara.register_driver :charai do |app|
  Charai::Driver.new(app, openai_configuration: config)
end

Capybara.register_driver :charai_headless do |app|
  Charai::Driver.new(app, openai_configuration: config, headless: true)
end

Please note that this driver required OpenAI service.

OpenAI

config = Charai::OpenaiConfiguration.new(
  model: 'gpt-4o',
  api_key: 'sk-xxxxxxxxxxx'
)

Azure OpenAI (Recommended)

config = Charai::AzureOpenaiConfiguration.new(
  endpoint_url: 'https://YOUR-APP.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-05-01-preview',
  api_key: 'aabbcc00112233445566'
)

Usage

Since this driver works with the OpenAI service, we can easily describe E2E test like below :)

before do
  Capybara.current_driver    = :charai
  Capybara.javascript_driver = :charai
  page.driver.additional_instruction = <<~MARKDOWN
  * このページは、3ペイン構造です。ユーザが仕事を探すためのページです。
  * 左ペインには仕事の絞り込みができるフィルター、中央ペインが仕事（求人）の一覧で、30件ずつ表示されます。
  * 左ペインにマウスを置いてスクロールしても、中央ペインはスクロールされません。一覧をスクロールしたいときには、中央ペインの座標を確認し、その中央にマウスを置いてスクロールしてください。
  * 右ペインは、広告エリアです。検索条件に応じた広告が表示されます。
  MARKDOWN
end

it 'should work' do
  page.driver << <<~MARKDOWN
  *

…

View on GitHub

Backport `Data.define` for Ruby 2.7, 3.0, 3.1

Yusuke Iwaki — Wed, 08 Mar 2023 04:17:00 +0000

Data.define is introduced by Zverok in Ruby 3.2.

New in Ruby 3.2 - Data.define

Brandon Weaver ・ Oct 6 '22 ・ 2 min read

This is really amazing and desired by most Rubyists. However, regardless to say, the function is not available in Ruby 2.7-3.1 applications. Some users (including me), at this moment, still uses older versions of Ruby with struggling to upgrade large Rails applications.

data_class_factory

I developed a backport of Data.define for Ruby 2.7, 3.0, 3.1.
https://rubygems.org/gems/data_class_factory

require 'data_class_factory'

Point = Data.define(:x, :y) do
  def norm
    Math.sqrt(x * x + y * y)
  end
end

p1 = Point.new(x: 3, y: 4)
p2 = Point.new(x: 3, y: 4)

p1 == p2 # => true
p1.norm # => 5

It covers most of the spec of original Data class introduced in Ruby 3.2.
https://github.com/YusukeIwaki/data_class_factory/blob/0.1.0/test/data_test.rb

Feedback is really welcome :)

I personally use this library on my Rails apps. So try this lib for your app, and let me know issues and feel free to feedback.
https://github.com/YusukeIwaki/data_class_factory/issues

Dart + GitHub Actions: Publish your command-line tools for Windows/macOS/Linux into GitHub Releases

Yusuke Iwaki — Sat, 02 Apr 2022 06:29:10 +0000

Motivation

Dart is very suitable for building quick-and-dirty CLI tools and share them among your collegue, since it produces a single binary with dart compile exe.

It would be useful if organization-internal tools are published in GitHub Releases like this:

Unfortunately, Dart doesn't support cross-compile at this moment and we have to compile the source code on each platform (Windows, macOS, Linux) before publishing the binaries, while Golang supports cross-compile and GoReleaser already made it easy to publish the binaries for each platform.

GitHub Actions provides Windows/macOS/Linux build environments for free and it would be really useful for compiling and publishing the CLI tools composed with Dart.

This article shows how to configure the GitHub Actions.

GitHub Actions for build

As many articles already says, only 4 step are required for compiling the Dart code.

- uses: actions/checkout@v2
- uses: dart-lang/setup-dart@v1
- run: dart pub get
- run: dart compile exe bin/mycli.dart -o mycli

One point to consider is that the output binary should be named as mycli .exe on Windows and mycli (without extension) on macOS/Linux.
We can solve the naming by defining the matrix like below:

jobs:
  compile:
    name: dart compile exe
    strategy:
      matrix:
        include:
          - runs-on: ubuntu-latest
            binary-name: mycli_linux_amd64
          - runs-on: macos-latest
            binary-name: mycli_macos_amd64
          - runs-on: windows-latest
            binary-name: mycli_windows.exe
    runs-on: ${{ matrix.runs-on }}
    steps:
      - uses: actions/checkout@v2
      - uses: dart-lang/setup-dart@v1
      - run: dart pub get
      - run: mkdir ${{ matrix.runs-on }}
      - run: dart compile exe bin/mycli.dart -o ${{ matrix.runs-on }}/${{ matrix.binary-name }}

GitHub Actions for release

https://docs.github.com/en/rest/reference/releases

According to this reference, GitHub Release requires 2 steps for publishing some binaries:

Create a release resource
- POST https://api.github.com/repos/{owner}/{repo}/releases
Upload the binaries into the release
- POST https://uploads.github.com/repos/{owner}/{repo}/releases/{release_id}/assets

Use action-gh-release for simple uploading

action-gh-release really makes it easy to do these steps.

- uses: softprops/action-gh-release@v1
  with:
    draft: true
    files: out/*

With this simple definition, GitHub Actions will upload all files in out/ into a new draft release.

    files: |
      out/mycli_linux
      out/mycli_macos
      out/mycli_windows

Also files can be specified to individual file list instead of glob expression.

Combine the build process with the release process using artifact

Each of 3 build actions generate a binary and release process should upload them all at once. The flow can be implemented with GitHub artifact.

name: Publish

on:
  push:
    tags:
      - '[0-9]+.[0-9]+.[0-9]+'

jobs:
  compile:
    name: dart compile exe
    strategy:
      matrix:
        include:
          - runs-on: ubuntu-latest
            binary-name: mycli_linux_amd64
          - runs-on: macos-latest
            binary-name: mycli_macos_amd64
          - runs-on: windows-latest
            binary-name: mycli_windows.exe
    runs-on: ${{ matrix.runs-on }}
    steps:
      - uses: actions/checkout@v2
      - uses: dart-lang/setup-dart@v1
      - run: dart pub get
      - run: mkdir ${{ matrix.runs-on }}
      - run: dart compile exe bin/mycli.dart -o ${{ matrix.runs-on }}/${{ matrix.binary-name }}
      - uses: actions/upload-artifact@v2
        with:
          name: bin-${{ matrix.runs-on }}
          path: ${{ matrix.runs-on }}

  release:
    needs: compile
    name: github release
    runs-on: ubuntu-latest
    steps:
      - uses: actions/download-artifact@v2
        with:
          name: bin-ubuntu-latest
          path: bin-linux
      - uses: actions/download-artifact@v2
        with:
          name: bin-macos-latest
          path: bin-macos
      - uses: actions/download-artifact@v2
        with:
          name: bin-windows-latest
          path: bin-windows
      - uses: softprops/action-gh-release@v1
        with:
          draft: true
          files: bin-*/*

This works expectedly like below:

and the artifacts are successfully uploaded to GitHub Releases! :)

System testing for Rails application without Capybara, using Puppeteer

Yusuke Iwaki — Fri, 03 Sep 2021 05:58:03 +0000

Refer this article instead if you can read Japanese sentences.
日本語を読める方はこちらの記事を見てください

Background

Rails introduced "system testing", that makes it easy to configure Capybara: acceptance test framework for Ruby. We may often use it not for "acceptance test" but for just UI testing of the Rails application with React/Vue or other rich JavaScript-powered features.

Capybara DSL is actually a little inaccurate for working with pages with rich JavaScript features.

Capybara has internally polling with interval 10msec for waiting for Elements available and actionable.
https://github.com/teamcapybara/capybara/blob/0468de5a810aae75ab9de20447e246c5c35473f0/lib/capybara/node/base.rb#L91
I don't know this is really the main cause, but we may often face flaky/unstable testcases in using Capybara.

On the other hand, Puppeteer or Playwright have a relyable solution for waiting for Elements: Page#waitForSelector and Page#waitForNavigation. (Playwright has more advanced feature of auto-waiting )

I actually ported the libraries for using them in Rails application.

Playwright for Ruby: https://playwright-ruby-client.vercel.app/
Puppeteer for Ruby: https://github.com/YusukeIwaki/puppeteer-ruby

However the relyable features cannot be used from Capybara DSL. So let's try to configure system tests without Capybara, and using Puppeteer.

What happens if Capybara is absent??

Since Capybara is a huge framework, it would be difficult for some users to imagine what happens without Capybara.

Actually Rails application without Capybara loses the functionalities

Launching test HTTP server on starting tests
Capybara::DSL
- Using visit or other methods raises NoMethodError instead of NotImplementedError
- System spec (on RSpec) or SystemTestCase (on MiniTest), which depends on Capybara::DSL

However rspec-rails still provides features like

Attaching type: :feature type: :system to example.metadata
Defining feature background scenario methods for feature spec

So actually all we have to prepare is

Launching test HTTP server
Performing automation with browsers
- This will be satisfied by Puppeteer

Launching test HTTP server without Capybara

Capybara is really kind that

finds available port
allows to configure host
allows to launch Webrick and Puma

However let's forget them at this moment, and assume that we only have to launch Puma with the baseURL http://127.0.0.1:3000

With referring the logics of Capybara preparing server and launching Puma server, then we can easily find the minimum configuration for launching test server like below:

RSpec.configure do |config|
  config.before(:suite) do
    # launching Rails server for system testing
    require 'rack/builder'
    testapp = Rack::Builder.app(Rails.application) do
      map '/__ping' do # debugging endpoint for heartbeat
        run ->(env) { [200, { 'Content-Type' => 'text/plain' }, ['OK']] }
      end
    end

    require 'rack/handler/puma'
    server_thread = Thread.new do
      Rack::Handler::Puma.run(testapp,
        Host: '127.0.0.1',
        Port: 3000,
        Threads: '0:4',
        workers: 0,
        daemonize: false,
      )
    end

    # Waiting for Rails server is ready, using Net::HTTP
    require 'net/http'
    require 'timeout'
    Timeout.timeout(3) do
      loop do
        puts Net::HTTP.get(URI("http://127.0.0.1:3000/__ping"))
        break
      rescue Errno::EADDRNOTAVAIL
        sleep 1
      rescue Errno::ECONNREFUSED
        sleep 0.1
      end
    end
  end

  # Configure puppeteer-ruby for automation
  config.around(type: :feature) do |example|
    Puppeteer.launch(channel: :chrome, headless: false) do |browser|
      @server_base_url = 'http://127.0.0.1:3000'
      @puppeteer_page = browser.new_page
      example.run
    end
  end
end

By putting this configuration into spec/support/integration_test_helper.rb or another place you prefer, now we can enjoy system testing with Puppeteer like below!!

As is already described, we cannot use system spec. Use feature spec (provided by rspec-rails) instead.

require 'rails_helper'

describe 'example' do
  let(:base_url) { @server_base_url }
  let(:page) { @puppeteer_page }

  let(:user) { FactoryBot.create(:user) }

  it 'can browse' do
    page.goto("#{base_url}/tests/#{user.id}")
    page.wait_for_selector('input', visible: true)
    page.type_text('input', 'hoge')
    page.keyboard.press('Enter')

    text = page.eval_on_selector('#content', 'el => el.textContent')
    expect(text).to include('hoge')
    expect(text).to include(user.name)
  end
end

A little refactoring for production use

The logic of launching test server, introduced in the previous section, is really straightforward. However most users would hesitate to copy/paste it into your own products because it's too dirty :(

No worry, here is a better code with more relyable Rack::Server.start to launch HTTP server, which is actually used in the implementation of rackup command

class RackTestServer
  def initialize(app:, **options)
    @options = options

    @options[:Host] ||= '127.0.0.1'
    @options[:Port] ||= 3000

    require 'rack/builder'
    @options[:app] = Rack::Builder.app(app) do
      map '/__ping' do
        run ->(env) { [200, { 'Content-Type' => 'text/plain' }, ['OK']] }
      end
    end
  end

  def base_url
    "http://#{@options[:Host]}:#{@options[:Port]}"
  end

  def start
    require 'rack/server'
    Rack::Server.start(**@options)
  end

  def ready?
    require 'net/http'
    begin
      Net::HTTP.get(URI("#{base_url}/__ping"))
      true
    rescue Errno::EADDRNOTAVAIL
      false
    rescue Errno::ECONNREFUSED
      false
    end
  end

  def wait_for_ready(timeout: 3)
    require 'timeout'
    Timeout.timeout(3) do
      sleep 0.1 until ready?
    end
  end
end

With this test server, the RSpec configuration file can be much shorter and simplified :)

RSpec.configure do |config|
  config.before(:suite) do
    # Launch Rails application
    test_server = RackTestServer.new(
      # options for Rack::Server
      # https://github.com/rack/rack/blob/2.2.3/lib/rack/server.rb#L173
      app: Rails.application,
      server: :puma,
      Host: '127.0.0.1',
      Port: 3000,
      daemonize: false,

      # options for Rack::Handler::Puma
      # https://github.com/puma/puma/blob/v5.4.0/lib/rack/handler/puma.rb#L84
      Threads: '0:4',
      workers: 0,
    )

    Thread.new { test_server.start }
    test_server.wait_for_ready
  end

  # Configure puppeteer-ruby for automation
  config.around(type: :feature) do |example|
    Puppeteer.launch(channel: :chrome, headless: false) do |browser|
      @server_base_url = 'http://127.0.0.1:3000'
      @puppeteer_page = browser.new_page
      example.run
    end
  end
end

Now, you can now get rid of Capybara from your app if you just want to launch HTTP server for testing :)

You can also use rack-test_server Gem instead of defining your own RackTestServer.

Using 🎭Playwright in Ruby/Rails

Yusuke Iwaki — Tue, 10 Aug 2021 14:25:06 +0000

🎭Playwright is a really awesome library for browser automation. Microsoft already published several language bindings.

TypeScript: https://github.com/microsoft/playwright
Java (or Kotlin): https://github.com/microsoft/playwright-java
Python: https://github.com/microsoft/playwright-python
C#: https://github.com/microsoft/playwright-dotnet

And also playwright-go is planned to be official.
https://github.com/microsoft/playwright/issues/6856

No Ruby client?

Ruby or Rails developers would like to use Playwright for acceptance testing or other use-cases, but it is not available at this moment.

Playwright team explicitly says there is no plan to release Ruby client.
https://github.com/microsoft/playwright/issues/5874

Ruby users should be happy with Playwright!!

It is clear that Ruby is not so famous as Python or TypeScript, but many Rails users still use it.

Rails already included an acceptance testing framework (called SystemTestCase). It uses Capybara and most users already noticed Capybara tends to produce flaky testcases.

Playwright provides very clever DOM selectors with auto-waiting feature, and useful utility functions. As Internet Explorer is already dead, no reason to keep using Selenium and Capybara.

Fortunately, Playwright is server-client architecture. We can use the useful features on any languages by just implementing a Playwright API client.

Try it on Ruby!!

I actually created a Playwright client for Ruby.
https://github.com/YusukeIwaki/playwright-ruby-client



Playwright.connect_to_playwright_server('ws://127.0.0.1:8080') do |playwright|
  playwright.chromium.launch do |browser|
    page = browser.new_page
    page.goto('https://github.com/YusukeIwaki')
    page.screenshot(path: './YusukeIwaki.png')
  end
end

Note that we have to launch Playwright server in advance, like this: npx playwright install && npx playwright run-server 8080

Try it on Rails!!

I also created a Capybara driver of Playwright.
https://github.com/YusukeIwaki/capybara-playwright-driver



Capybara.register_driver(:playwright) do |app|
  Capybara::Playwright::Driver.new(app, browser_type: :firefox, headless: false)
end
Capybara.default_max_wait_time = 15
Capybara.default_driver = :playwright

Capybara DSL is supported, and we can also use Playwright-native features such as recording video during testing.
https://playwright-ruby-client.vercel.app/docs/article/guides/rails_integration#available-functions-and-limitations

Conclusion

Playwright's architecture is great, and we Ruby/Rails users can use Playwright features by developing our own Playwright client API library.

Unfortunately Microsoft doesn't make effort for Ruby users, and difficult to ask for official support for Ruby.

Using 🎭Playwright in python:alpine

Yusuke Iwaki — Tue, 10 Aug 2021 13:41:26 +0000

playwright-python is a really awesome library for browser automation.

However it doesn't support running on Alpine Linux.
https://github.com/microsoft/playwright-python/issues/234

Most web developers would use Alpine based Docker images, and actually playwright-python internally has a capability of running scripts in Alpine.

Why playwright-python cannot run on Alpine?

playwright-python comes with a browser downloader. As we know, browsers requires many dependencies. The deps are not compatible with Alpine.

This might be a main reason why playwright-python deny running on Alpine.

By the way, we already know Selenium Remote driver (Selenium Grid), it separates script runner and browser environments communicating with HTTP port (default: 4444).

How to separate script runner and browsers

playwright-python is just a Python client library of Playwright. Playwright server communicates with browser via WebSocket or other protocols, and playwright-python communicates with the server via pipe by default.

However playwright-python already introduced WebSocketTransport for the API BrowserType#connect() in Playwright v1.11 and later. It can be also used for Playwright connection between client and server.

Try it!

I created a tiny library.
https://github.com/YusukeIwaki/playwright-python-remote

This library provides a Playwright initializer (ContextManager) just replacing PipeTransport with WebSocketTransport.
https://github.com/YusukeIwaki/playwright-python-remote/blob/main/playwright_remote/sync_api/sync_playwright_remote.py

Playwright Server can be prepared very easily with Docker like this:



FROM mcr.microsoft.com/playwright:focal

WORKDIR /root
RUN npm install playwright@1.12.3 && ./node_modules/.bin/playwright install
CMD ["./node_modules/.bin/playwright", "run-server"]

Now we can enjoy playwright-python on Alpine:



from playwright_remote.sync_api import sync_playwright_remote

with sync_playwright_remote('ws://127.0.0.1:8080/ws') as playwright:
  browser = playwright.chromium.launch()
  try:
    page = browser.new_page()
    page.goto('https://github.com/YusukeIwaki')
    page.screenshot(path='YusukeIwaki.png')
  finally:
    browser.close()

I also tried it on Heroku: https://zenn.dev/yusukeiwaki/scraps/3ad5f8ed536720 (Sorry, Japanese only...)

Conclusion

playwright-python is really awesome, it runs on Alpine actually.

Note that WebSocketTransport (introduced in this article) is internal class and can be changed in future releases.

Just expect the feature is officially supported. Let's upvote
https://github.com/microsoft/playwright/issues/4687 👍