DEV Community: Seryl Lns

Building a Rails Engine #16 --Publishing to RubyGems & Retrospective

Seryl Lns — Thu, 09 Apr 2026 12:06:10 +0000

Publishing to RubyGems & Retrospective

From bundle gem to gem push: looking back at 14 articles, 20 components, and the lessons learned building a Rails engine from scratch with TDD.

Context

This is the final article in the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 14, we added Dry Run mode -- the last safety net before data touches the database.

We started this series with a question: why do we keep rebuilding the same import workflow in every Rails app? Fourteen articles later, we have a published gem that answers it. This article covers the last mile -- publishing to RubyGems -- then steps back to look at what we built, what we learned, and what we would do differently.

Publishing the gem

The gemspec

The interesting parts of the gemspec are not the metadata -- they are the constraints:

# data_porter.gemspec
Gem::Specification.new do |spec|
  spec.name = "data_porter"
  spec.version = DataPorter::VERSION
  spec.required_ruby_version = ">= 3.2.0"

  spec.metadata["rubygems_mfa_required"] = "true"

  spec.add_dependency "csv"
  spec.add_dependency "phlex", ">= 1.0"
  spec.add_dependency "rails", ">= 7.0"
  spec.add_dependency "store_model", ">= 2.0"
  spec.add_dependency "turbo-rails", ">= 1.0"
end

rubygems_mfa_required enforces multi-factor authentication for publishing -- a standard for any serious open-source gem. required_ruby_version at >= 3.2.0 excludes unmaintained Ruby versions. Runtime dependencies are intentionally wide (>= 1.0, >= 7.0) to avoid locking host apps to specific versions.

The spec.files filter excludes dev files (spec/, bin/, .github/) so the published gem only contains production code. Nobody wants to download 2 MB of specs when installing a gem.

Versioning

DataPorter follows semantic versioning:

0.1.0: first release. The 0.x signals that the API may still evolve.
0.x.y: each new feature increments minor, each bugfix increments patch.
1.0.0: comes when the API is stabilized and battle-tested in production across multiple apps.

The version number lives in a single file:

# lib/data_porter/version.rb
module DataPorter
  VERSION = "0.1.0"
end

One place to update. The gemspec reads it via require_relative. The CHANGELOG references it. The Git tag matches it. No duplication.

The release workflow

# 1. Update version
# lib/data_porter/version.rb -> VERSION = "0.1.0"

# 2. Update CHANGELOG
# CHANGELOG.md -> ## [0.1.0] - 2026-02-06

# 3. Commit, tag, push
git add -A && git commit -m "Release v0.1.0"
git tag v0.1.0
git push origin master --tags

# 4. Build and push
gem build data_porter.gemspec
gem push data_porter-0.1.0.gem

Or, if the Rakefile includes bundler/gem_tasks:

bundle exec rake release

This single command builds, tags, pushes to Git, and pushes to RubyGems -- guaranteeing the tag and the gem stay in sync.

Documentation

A gem without documentation is a gem nobody will use. DataPorter relies on three layers:

The README: entry point. Install in one command (rails generate data_porter:install), a 15-line Target example, the three-step workflow diagram. A developer should understand what the gem does and install it in under 5 minutes.

The CHANGELOG: every release documented with what changed, what was added, what broke. Keep a Changelog format -- a standard the Ruby community knows.

Inline comments: every public method documented with YARD. The DSL is the critical part -- column, sources, csv_mapping, persist need examples, because that is what users will read most.

What we built

Here is the complete list of components that make up DataPorter, in the order we built them:

#	Component	Role
1	Engine + isolate_namespace	Gem structure, namespace isolation
2	Configuration DSL	`DataPorter.configure`, defaults, `context_builder`
3	StoreModels (ImportRecord, Error, Report)	Typed JSONB structures without extra tables
4	TypeValidator	Type validation (email, phone, url) on columns
5	Target DSL	`label`, `model`, `columns`, `sources`, `persist`
6	Registry	Auto-discovery and resolution of targets
7	Source::Base + Source::CSV	Source abstraction, CSV parsing with mapping
8	DataImport model	ActiveRecord, enum status, polymorphic user
9	Orchestrator	Coordinates parse/import, per-record error handling
10	RecordValidator	Generic validations (required, type)
11	ParseJob + ImportJob	Background processing via ActiveJob
12	Broadcaster + ImportChannel	Real-time progress via ActionCable
13	6 Phlex components	StatusBadge, SummaryCards, PreviewTable, ProgressBar, ResultsSummary, FailureAlert
14	Stimulus controller	Client-side progress bar animation
15	ImportsController	Dynamic inheritance, 7 actions, Turbo integration
16	Install generator	Migration, initializer, routes, importers directory
17	Target generator	Target scaffolding with column parsing
18	Source::JSON	Import from JSON file or raw text
19	Source::API	Import from HTTP endpoint with auth and params
20	Dry Run	Transaction + rollback, enriches records with DB errors

Twenty components. Each with its specs. Each with an article explaining why it exists and how it works.

Lessons learned

TDD without a dummy app

The most consequential decision of the series: testing the engine without creating a Rails application in spec/dummy/. A 60-line spec_helper.rb that bootstraps in-memory SQLite, configures load paths, and stubs ApplicationController. It works, and it works well -- the full suite runs in under a second.

The unexpected benefit: this constraint forces every component to stay decoupled. If a component needs a router to be tested, that is a signal it is too tightly coupled to the framework. Structural controller tests (verifying inheritance, callbacks, method signatures) felt strange at first. In hindsight, they test exactly what the gem owns -- the wiring -- and leave integration testing to the host app.

The trap to avoid: duplication between the schema in spec_helper.rb and the migration template. If the two diverge, tests pass but the generated migration does not match what was tested.

StoreModel gotchas

StoreModel is powerful, but it has its subtleties:

Dirty tracking: when you modify an object inside a store_model attribute, ActiveRecord does not detect the change. You can set data_import.records.first.status = "complete" and call save -- nothing gets persisted. The fix: call records_will_change! before modifying, or reassign the entire attribute.

Serialization round-trip: symbol keys become string keys after save/reload. { name: "Alice" } comes back as { "name" => "Alice" }. You need to know this and code accordingly -- either always use string keys, or call symbolize_keys on the way out. DataPorter does the latter in ImportRecord#attributes.

SQLite vs PostgreSQL: in tests, StoreModel columns are text. In production, they are jsonb. StoreModel handles the difference transparently, but certain JSONB queries (indexes, contains) cannot be tested in SQLite. An acceptable tradeoff for the speed of the feedback loop.

Phlex in an engine: `plain` vs `text`

A Phlex-specific trap: to emit raw text inside an element, you must use plain (not text). In earlier Phlex versions, text existed but was renamed. If you use text with a recent version, you get a cryptic NoMethodError.

The other subtlety: calling super() in every component's initialize. Phlex requires it, and forgetting it produces silent errors or empty renders.

Testing patterns: controllers, channels, JS

Testing JavaScript from Ruby by reading the file as text and asserting on strings -- it sounds hacky. In practice, it catches the most common bug class in an engine: misalignment between Ruby and JS code. The channel is called DataPorter::ImportChannel in Ruby and "DataPorter::ImportChannel" in JS. If one changes and the other does not, the test fails. For a single 30-line Stimulus file, that beats adding Jest and node_modules to the project.

Structural controller tests (_process_action_callbacks, instance_method, superclass) form a contract: the gem guarantees the controller has the right shape. The host app guarantees it behaves correctly in context. A clean separation of responsibilities.

What is next

DataPorter 0.1.0 covers the standard workflow. Here is what could come in future versions:

Batch imports: for 100k+ line files, import in batches of 1000 with insert_all instead of create! per record. This requires rethinking the persist contract -- instead of one record at a time, the target would receive a batch.

Streaming progress: replace ActionCable with Server-Sent Events (SSE) for apps that do not need bidirectional WebSocket. Lighter, no Redis dependency.

Custom validators: let targets declare validators with a DSL:

columns do
  column :email, type: :email, required: true, validate: ->(val) {
    "already exists" if User.exists?(email: val)
  }
end

Export: the reverse path. If we can parse and validate records, we can serialize them to CSV/JSON. The Target already has all the information needed (columns, types, labels).

Final reflection

Building DataPorter was an exercise in discipline as much as code. The method -- Taskmaster for planning, TDD for implementation, one article to document each step -- forces explicit decisions. No "we will figure it out later". Every component exists because a test demands it, and every test exists because a behavior was specified.

The choice to skip the dummy app was a gamble. It paid off: tests are fast, components are decoupled, and the gem is testable without Rails infrastructure. But it has a cost -- some integration bugs will only surface in the host app. That is an accepted tradeoff: the gem tests its wiring, the host app tests its behavior.

StoreModel, Phlex, Stimulus -- each dependency brought its share of surprises. StoreModel's dirty tracking, Phlex's plain vs text, Stimulus's double-dash naming for engines. These gotchas appear in no documentation. They appear when a test fails at 11 PM and you read the gem's source code to understand why. That is the real advantage of TDD: you discover problems in the terminal, not in production.

DataPorter is now a published gem on RubyGems. One bundle add data_porter, one rails generate data_porter:install, a 15-line Target, and any Rails app has a complete import system with preview, validation, real-time progress, and dry run.

That was the plan from the start. It took 16 articles to get there.

This is part 16 of the series "Building DataPorter - A Data Import Engine for Rails". Previous: ERB Views Meet Phlex Components

GitHub: SerylLns/data_porter | RubyGems: data_porter

Building a Rails Engine #15 -- ERB Views Meet Phlex Components

Seryl Lns — Tue, 31 Mar 2026 13:33:36 +0000

ERB View Templates: Composing Phlex Components

Phlex components are pure Ruby. ERB templates are what the browser actually sees. The glue between the two is a single method: .call.

Context

This is part 16 of the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 9, we built six Phlex components -- StatusBadge, SummaryCards, PreviewTable, ProgressBar, ResultsSummary, FailureAlert. In part 10, we built the controller and routes. But we never connected the two.

The engine has a fully wired backend and a complete component library, but visiting /imports crashes with ActionView::MissingTemplate. The app/views/data_porter/imports/ directory is empty. The controller sets instance variables, the components know how to render HTML, but there is no view template to orchestrate them.

In this article, we create three ERB templates -- index, new, and show -- that compose the existing Phlex components into full pages.

The missing attachment

Before touching views, there is a bug to fix. The CSV and JSON sources both call @data_import.file.download, but the DataImport model has no file attachment declared:

# app/models/data_porter/data_import.rb
module DataPorter
  class DataImport < ActiveRecord::Base
    self.table_name = "data_porter_imports"

    has_one_attached :file

    belongs_to :user, polymorphic: true, optional: true

Two changes here. has_one_attached :file makes ActiveStorage available on the model. And optional: true on the belongs_to -- in Rails 5+, belongs_to associations are required by default. An engine should not enforce that constraint; the host app decides whether imports require a user.

Adding has_one_attached has a cascade effect on the test setup. ActiveStorage needs its tables (blobs, attachments, variant_records), a configured service, and a running Rails application to initialize the engine. Our previous spec_helper -- 15 lines of manual requires and an in-memory SQLite connection -- is not enough anymore.

Upgrading the test infrastructure

The spec_helper needed a fundamental change: bootstrapping a real Rails application instead of manually requiring individual components.

# spec/spec_helper.rb
ENV["RAILS_ENV"] = "test"

require "rails"
require "active_record"
require "active_job"
require "action_controller"
require "action_cable"
require "active_storage/engine"

module TestApp
  class Application < Rails::Application
    config.load_defaults Rails::VERSION::STRING.to_f
    config.eager_load = false
    config.active_storage.service = :test
    config.active_storage.service_configurations = {
      test: { service: "Disk", root: Dir.tmpdir }
    }
    config.active_job.queue_adapter = :test
    config.secret_key_base = "test_secret"
    config.root = File.expand_path("..", __dir__)
  end
end

require "data_porter"

Rails.application.initialize!

ActiveRecord::Base.establish_connection(adapter: "sqlite3", database: ":memory:")

The key insight: Rails.application.initialize! must run after the connection is established, and before the schema is defined. ActiveStorage's engine hooks into the initialization process to register its models and set up the attachment macros. Without initialize!, has_one_attached raises NoMethodError because ActiveStorage::Attached::Model is never included into ActiveRecord::Base.

The schema definition gains three new tables for ActiveStorage -- active_storage_blobs, active_storage_attachments, and active_storage_variant_records -- mirroring what rails active_storage:install creates in a host app.

We also need a User stub and a config/database.yml for Rails to find during initialization:

# config/database.yml
test:
  adapter: sqlite3
  database: ":memory:"

# spec/spec_helper.rb (after schema)
class User < ActiveRecord::Base; end unless defined?(User)

The User model was previously not needed because without a full Rails app, the polymorphic belongs_to never tried to resolve the User constant. With initialize!, Rails activates the belongs_to presence validation, and any test that creates a DataImport with user_type: "User" triggers a NameError. The stub class fixes it cleanly.

One more piece: engine routes must be mounted in the test app after initialization:

Rails.application.routes.draw do
  mount DataPorter::Engine, at: "/data_porter"
end

This is crucial for the view specs. Without it, the engine's URL helpers (import_path, imports_path) cannot resolve paths because the mount point is unknown.

Rendering Phlex in ERB

DataPorter uses Phlex without phlex-rails. This means no render helper for Phlex components in ERB templates. Instead, we call .call directly, which returns an HTML string, and wrap it in raw to prevent double-escaping:

<%= raw DataPorter::Components::StatusBadge.new(status: @import.status).call %>

This is explicit and requires no gems, no monkey-patching, no initializer configuration. The tradeoff is that every component call is slightly verbose. But for a gem that ships views, this is a feature: the host app does not need to install phlex-rails, and there is no risk of version conflicts between the gem's Phlex setup and the host app's.

The index template

The index page lists all imports with their status, target, source type, and creation date:

<%# app/views/data_porter/imports/index.html.erb %>
<div class="data-porter">
  <div class="dp-header">
    <h1 class="dp-title">Imports</h1>
    <%= link_to "New Import", new_import_path, class: "dp-btn dp-btn--primary" %>
  </div>

  <table class="dp-table">
    <thead>
      <tr>
        <th>ID</th>
        <th>Target</th>
        <th>Source</th>
        <th>Status</th>
        <th>Created</th>
        <th></th>
      </tr>
    </thead>
    <tbody>
      <% @imports.each do |import| %>
        <tr>
          <td><%= import.id %></td>
          <td><%= import.target_key %></td>
          <td><%= import.source_type %></td>
          <td><%= raw DataPorter::Components::StatusBadge.new(status: import.status).call %></td>
          <td><%= import.created_at&.strftime("%Y-%m-%d %H:%M") %></td>
          <td><%= link_to "View", import_path(import), class: "dp-link" %></td>
        </tr>
      <% end %>
    </tbody>
  </table>
</div>

Each row includes a StatusBadge component rendered inline. The outer <div class="data-porter"> scopes all styles. The URL helpers (new_import_path, import_path) are resolved through the engine's isolated namespace -- no data_porter. prefix needed inside engine views.

The new import form

The form uses form_with with three fields: a target select (populated from DataPorter::Registry.available), a source type select (from DataPorter.configuration.enabled_sources), and a file upload field. The key detail:

<%= form_with model: @import, url: imports_path, class: "dp-form" do |f| %>
  <%= f.select :target_key,
        @targets.map { |t| [t[:label], t[:key]] },
        { prompt: "Select a target..." }, class: "dp-select" %>
  <%= f.file_field :file, class: "dp-file-input" %>
  <%= f.submit "Start Import", class: "dp-btn dp-btn--primary" %>
<% end %>

The url: imports_path is necessary because the model is a DataPorter::DataImport, and Rails would try to generate a data_porter_data_import_path without the explicit URL. Inside engine views, explicit URLs for form actions avoid routing surprises.

The show template: status-driven composition

The show page is where all the Phlex components come together. The key design: the template renders different components based on the import's current status:

<%# app/views/data_porter/imports/show.html.erb %>
<div class="data-porter">
  <div class="dp-header">
    <h1 class="dp-title">
      <%= @target._label %> Import #<%= @import.id %>
    </h1>
    <%= raw DataPorter::Components::StatusBadge.new(status: @import.status).call %>
  </div>

  <% if @import.parsing? || @import.importing? || @import.dry_running? %>
    <%= raw DataPorter::Components::ProgressBar.new(import_id: @import.id).call %>
  <% end %>

  <% if @import.previewing? %>
    <%= raw DataPorter::Components::SummaryCards.new(report: @import.report).call %>
    <%= raw DataPorter::Components::PreviewTable.new(
          columns: @target._columns,
          records: @records
        ).call %>

    <div class="dp-actions">
      <%= button_to "Confirm", confirm_import_path(@import),
            method: :post, class: "dp-btn dp-btn--primary" %>
      <% if @target._dry_run_enabled %>
        <%= button_to "Dry Run", dry_run_import_path(@import),
              method: :post, class: "dp-btn dp-btn--secondary" %>
      <% end %>
      <%= button_to "Cancel", cancel_import_path(@import),
            method: :post, class: "dp-btn dp-btn--danger" %>
    </div>
  <% end %>

  <% if @import.completed? %>
    <%= raw DataPorter::Components::ResultsSummary.new(report: @import.report).call %>
  <% end %>

  <% if @import.failed? %>
    <%= raw DataPorter::Components::FailureAlert.new(report: @import.report).call %>
    <div class="dp-actions">
      <%= button_to "Retry", parse_import_path(@import),
            method: :post, class: "dp-btn dp-btn--primary" %>
    </div>
  <% end %>
</div>

Five states, five rendering paths:

Status	Components rendered
`parsing`, `importing`, `dry_running`	ProgressBar
`previewing`	SummaryCards + PreviewTable + action buttons
`completed`	ResultsSummary
`failed`	FailureAlert + Retry button
`pending`	Header only (waiting for job to start)

The Dry Run button appears only when the target has enabled it (@target._dry_run_enabled). This is the DSL flag from part 14 surfacing in the UI -- targets opt in to dry run, and the view respects that choice without any extra configuration.

The action buttons use button_to which generates a mini-form with a POST request. This matches the route design from part 10: all member actions (confirm, cancel, dry_run, parse) are POST endpoints. No JavaScript needed for these actions -- they are standard form submissions that Turbo handles automatically.

The CSS stylesheet

DataPorter ships a plain CSS file with styles for all dp-* classes. No build step, no Tailwind compilation, no PostCSS pipeline. Every class is scoped under .data-porter or prefixed with dp- -- the host app's styles cannot accidentally override DataPorter's components, and DataPorter's styles cannot leak out.

The full stylesheet covers tables, badges, cards, progress bars, forms, buttons, alerts, and results. Host apps can override any of these styles or ignore the stylesheet entirely and provide their own -- the dp- prefix convention means they always have a clean selector to target.

Testing views in a Rails 8 engine

Testing ERB templates in a Rails engine without a dummy app is not well documented. Rails 8 changed how ActionView::Base works -- you need with_empty_template_cache to create a usable view class, and the view needs both the engine's URL helpers and the main app's URL helpers to resolve paths correctly.

The solution lives in a shared helper:

# spec/support/view_test_helper.rb
module ViewTestHelper
  VIEW_CLASS = ActionView::Base.with_empty_template_cache
  VIEW_CLASS.include DataPorter::Engine.routes.url_helpers
  VIEW_CLASS.include Rails.application.routes.url_helpers

  def build_view(assigns = {})
    lookup = ActionView::LookupContext.new(view_paths)
    ctrl = DataPorter::ImportsController.new
    ctrl.request = ActionDispatch::TestRequest.create
    ctrl.default_url_options = { host: "localhost" }

    view = VIEW_CLASS.new(lookup, assigns, ctrl)
    view.default_url_options = { host: "localhost" }
    view
  end

  def view_paths
    [File.expand_path("../../app/views", __dir__)]
  end
end

Three things are critical here:

ActionView::Base.with_empty_template_cache -- In Rails 8, ActionView::Base.new raises NotImplementedError asking for compiled_method_container. The with_empty_template_cache class method creates a subclass that has the right template caching setup.

Both route helper modules -- The engine's URL helpers (import_path, imports_path) resolve paths within the engine. The main app's URL helpers provide the mount point (data_porter_path) that the engine needs to construct full URLs. Without the main app helpers, named routes raise NoMethodError: undefined method 'data_porter_path'.

DataPorter::ImportsController.new as the controller -- The view delegates _routes resolution to its controller. Using ActionController::Base.new fails because it has no _routes for the engine namespace. Using the actual ImportsController -- which inherits the engine's route set -- makes all URL helpers work correctly.

The specs themselves are straightforward:

RSpec.describe "data_porter/imports/show.html.erb" do
  include ViewTestHelper

  context "when previewing" do
    let(:status) { :previewing }

    it "renders the summary cards" do
      expect(html).to include("dp-summary-cards")
    end

    it "renders the preview table" do
      expect(html).to include("dp-preview-table")
    end

    it "shows confirm button" do
      expect(html).to include("Confirm")
    end
  end

  context "when failed" do
    it "renders the failure alert" do
      expect(html).to include("dp-alert")
    end

    it "shows retry button" do
      expect(html).to include("Retry")
    end
  end
end

Each context sets a different import status and verifies that the correct components and buttons are present. The assertions check for CSS class names (dp-summary-cards, dp-alert) rather than exact HTML structure, making them resilient to markup changes while still catching the important behavior: "this component is rendered for this status".

Decisions & tradeoffs

Decision	We chose	Over	Because
Phlex integration	`raw component.call` in ERB	phlex-rails gem with `render` helper	No extra dependency; explicit about what is happening; host app does not need phlex-rails installed
Template format	ERB wrappers composing Phlex	Pure Phlex page components	ERB is familiar to every Rails developer; forms and URL helpers work naturally; Phlex handles the complex rendering
Status-driven rendering	`if @import.parsing?` conditionals	Separate templates per status, Turbo Frame swapping	Simple, readable, works without JavaScript; one template to understand
CSS approach	Plain CSS file, no build step	Tailwind build, CSS-in-JS, Sass	Zero dependencies; host app can use any CSS pipeline; works out of the box
View test approach	`ActionView::Base.with_empty_template_cache` + engine controller	Dummy app, request specs, Capybara	Fast, no extra infrastructure; tests the template rendering directly
Form URL	Explicit `url: imports_path`	Let Rails infer from model	Avoids routing surprises with engine-namespaced models

Recap

has_one_attached :file was missing from DataImport -- a bug that would crash CSV and JSON imports at runtime. Adding it required upgrading the test infrastructure to bootstrap a full Rails application with ActiveStorage.
ERB templates compose Phlex components via raw component.call -- no phlex-rails dependency needed. The show template renders different components based on import status, making the page dynamic without JavaScript.
Plain CSS with dp-* prefixed classes ships with the gem and works out of the box. Host apps can override or replace it entirely.
View testing in Rails 8 requires ActionView::Base.with_empty_template_cache, both engine and main app URL helpers, and the engine's own controller for proper route resolution. The ViewTestHelper encapsulates this setup in a reusable module.
22 view specs cover all three templates and all status-driven rendering paths, running in under a second with no browser or dummy app.

Next up

With the views in place, DataPorter is feature-complete. In the next and final article, we step back for a showcase and retrospective -- screenshots of the full workflow in a real app, a walkthrough from upload to import, and reflections on what the 16-article journey taught us about building Rails engines.

This is part 16 of the series "Building DataPorter - A Data Import Engine for Rails". Dry Run: Validate Before You Import | Next: Previous: Publishing & Retrospective

GitHub: SerylLns/data_porter | RubyGems: data_porter

Building a Rails Engine #14 -- Dry Run: Validate Before You Import

Seryl Lns — Wed, 25 Mar 2026 13:57:47 +0000

Dry Run: Validate Before You Import

The preview catches column errors. The dry run catches database errors. Two safety nets, two levels of confidence.

Context

This is part 14 of the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 13, we detailed the testing strategy: in-memory SQLite, structural controller specs, anonymous target classes, and a spec_helper that bootstraps just enough Rails to cover every layer.

We now have a complete import pipeline: parse the file, preview the records, confirm, persist. But there is a gap between the preview and the real import. The preview validates the data -- required fields, types, formats. It does not validate what happens when that data hits the database. A uniqueness constraint, a foreign key violation, a custom model validation that queries other tables -- none of these surface until persist is actually called. By then, the import is running for real.

In this article, we build a dry run mode that bridges this gap. It runs the full persist logic inside a transaction, captures any database-level errors on each record, then rolls back. The user sees exactly which records would fail before committing to the import.

Why two validation layers

The preview phase runs the RecordValidator against column definitions. It catches structural problems: "this field is required and it is empty", "this field should be an email but it is not". These are fast, stateless checks that do not touch the database.

But many real-world validations are stateful. A validates_uniqueness_of :email on the User model requires a database query. A belongs_to :company with a foreign key constraint requires the company to exist. A custom validation that checks if: -> { some_scope.exists? } requires the full ActiveRecord context. None of these can run during preview because there is no model instance, no transaction, no database connection in the validation path.

The dry run fills this gap. It calls the target's persist method -- the same method that the real import uses -- but captures exceptions instead of letting them propagate. Each record gets annotated with its result: passed or failed, with the error message attached.

The `dry_run_enabled` DSL flag

Not every import needs a dry run. A simple CSV-to-table import with no uniqueness constraints might not benefit from the overhead. We make it opt-in at the target level:

# app/importers/user_import.rb
class UserImport < DataPorter::Target
  label "Users"
  model_name "User"
  dry_run_enabled

  columns do
    column :email, type: :email, required: true
    column :name, type: :string, required: true
  end

  def persist(record, context:)
    User.create!(record.attributes)
  end
end

The dry_run_enabled class method is a simple flag on the Target DSL:

# lib/data_porter/target.rb
class << self
  attr_reader :_dry_run_enabled

  def dry_run_enabled
    @_dry_run_enabled = true
  end
end

No arguments, no block, no configuration. Either the target supports dry run or it does not. The controller checks target_class._dry_run_enabled to decide whether to show the "Dry Run" button on the preview page.

The `dry_running` status

The DataImport enum needed a new state. The import transitions from previewing to dry_running while the dry run executes, then back to previewing when it completes:

# app/models/data_porter/data_import.rb
enum :status, {
  pending: 0,
  parsing: 1,
  previewing: 2,
  importing: 3,
  completed: 4,
  failed: 5,
  dry_running: 6
}

The value 6 is appended at the end rather than inserted in logical order. This is intentional -- existing records in production have integer status values. Inserting dry_running at position 3 would shift importing, completed, and failed, corrupting every existing import. Enums with integer backing must be append-only.

The state flow is: previewing -> dry_running -> previewing. The dry run is not a terminal state. It enriches the records with database-level feedback and returns to the preview so the user can review the results and decide whether to proceed with the real import.

The `dry_run!` flow

The Orchestrator gains a third public method alongside parse! and import!:

# lib/data_porter/orchestrator.rb
def dry_run!
  @data_import.dry_running!
  run_dry_run_records
  @data_import.update!(status: :previewing)
  build_report
rescue StandardError => e
  handle_failure(e)
end

The structure mirrors parse! and import!: transition to the working status, do the work, transition to the result status, rebuild the report. The rescue catches catastrophic failures and transitions to failed with an error report.

The real work happens in run_dry_run_records:

def run_dry_run_records
  records = @data_import.records
  importable = records.select(&:importable?)
  context = build_context

  importable.each do |record|
    dry_run_record(record, context)
  end

  @data_import.records_will_change!
  @data_import.update!(records: records)
end

def dry_run_record(record, context)
  @target.persist(record, context: context)
  record.dry_run_passed = true
rescue StandardError => e
  record.dry_run_passed = false
  record.add_error(e.message)
end

For each importable record, we call the target's actual persist method. If it succeeds, dry_run_passed is set to true. If it raises -- an ActiveRecord::RecordInvalid, a constraint violation, any exception -- we capture the message on the record and mark it as failed. The import data is never committed because the dry run operates at the record level, catching errors individually.

The dry_run_passed attribute on ImportRecord is a simple boolean:

# lib/data_porter/store_models/import_record.rb
attribute :dry_run_passed, :boolean, default: false

After the dry run, the preview table can show a green check or a red cross next to each record, along with the specific error message for failures. The user gets a precise map of what will work and what will not.

The StoreModel dirty tracking gotcha

There is a subtle but critical line in run_dry_run_records:

@data_import.records_will_change!
@data_import.update!(records: records)

Why records_will_change! before update!? The answer lies in how ActiveRecord tracks changes on complex attributes.

StoreModel attributes are serialized to JSON and stored in a text (or JSONB) column. When you modify an object in place -- setting record.dry_run_passed = true on a record that already exists in the records array -- ActiveRecord does not detect the change. From its perspective, the records attribute still points to the same Ruby array at the same memory address. The serialized value has changed, but ActiveRecord's dirty tracking compares object identity, not serialized content.

Without records_will_change!, the update! call would see "records has not changed" and skip the column in the SQL UPDATE. The dry run results would be computed correctly in memory but never persisted to the database. The user would see no change on the preview page.

records_will_change! explicitly marks the attribute as dirty, forcing ActiveRecord to include it in the next save. This is a well-known pattern with serialized attributes, but it is easy to forget -- and the failure mode is silent. The data looks correct in the current process, the tests that do not reload from the database pass, and only the production user sees stale results.

This is one of those bugs that TDD catches early. The spec reloads the import from the database and checks dry_run_passed on the reloaded records:

it "marks records as dry_run_passed on success" do
  DataPorter::Orchestrator.new(import.reload).dry_run!
  import.reload.records.each do |record|
    expect(record.dry_run_passed).to be true
  end
end

The import.reload forces a fresh read from SQLite. Without records_will_change!, this spec fails -- dry_run_passed is still false in the database even though it was set to true in memory.

Job, controller, and route

The wiring follows the exact same pattern as parse! and import!: a thin job delegates to the Orchestrator, a controller action enqueues it, a member route exposes it.

# app/jobs/data_porter/dry_run_job.rb
class DryRunJob < ActiveJob::Base
  queue_as { DataPorter.configuration.queue_name }

  def perform(import_id)
    data_import = DataImport.find(import_id)
    Orchestrator.new(data_import).dry_run!
  end
end

# app/controllers/data_porter/imports_controller.rb
def dry_run
  DataPorter::DryRunJob.perform_later(@import.id)
  redirect_to import_path(@import)
end

POST triggers the job, redirect back to the show page where ActionCable pushes progress updates. The view conditionally shows the "Dry Run" button only when target_class._dry_run_enabled is true and the import is in previewing status.

Testing

The dry run specs follow the series' established patterns -- anonymous target classes, registry cleanup, and database round-trip assertions:

RSpec.describe "Dry Run" do
  let(:target_class) do
    klass = Class.new(DataPorter::Target) do
      label "Guests"
      model_name "Guest"
      dry_run_enabled

      columns do
        column :first_name, type: :string, required: true
        column :last_name, type: :string
      end
    end
    klass.define_method(:persist) do |record, context:|
      record
    end
    klass
  end

  describe "Orchestrator#dry_run!" do
    it "transitions to previewing after dry run" do
      DataPorter::Orchestrator.new(import.reload).dry_run!
      expect(import.reload.status).to eq("previewing")
    end

    it "marks records as dry_run_passed on success" do
      DataPorter::Orchestrator.new(import.reload).dry_run!
      import.reload.records.each do |record|
        expect(record.dry_run_passed).to be true
      end
    end

    it "captures errors from failing persist" do
      # Target that raises ActiveRecord::RecordInvalid
      DataPorter::Orchestrator.new(failing_import.reload).dry_run!

      record = failing_import.reload.records.first
      expect(record.dry_run_passed).to be false
      expect(record.errors_list.map(&:message)).to include(match(/Validation failed/))
    end
  end
end

The failing target class simulates a database-level error by raising ActiveRecord::RecordInvalid in persist. The spec verifies that the error is captured on the record, that dry_run_passed is false, and that the import still transitions to previewing -- not failed. A record-level error is expected operational feedback, not a catastrophic failure.

Decisions & tradeoffs

Decision	We chose	Over	Because
Opt-in flag	`dry_run_enabled` on Target DSL	Always-on dry run	Not every import benefits from the overhead; simple imports can skip it
Status value	Append `dry_running: 6` at the end of the enum	Insert in logical order	Integer-backed enums must be append-only to avoid corrupting existing data
Dirty tracking	Explicit `records_will_change!`	Reassigning the array (`self.records = records.dup`)	More explicit about intent; avoids unnecessary array duplication; documents the StoreModel gotcha
Error boundary	Per-record rescue in `dry_run_record`	Wrapping all records in a single begin/rescue	One failing record should not prevent the others from being validated

Recap

The dry run bridges the gap between preview (column-level validation) and real import (database-level validation), giving users a complete picture before any data is committed.
The dry_run_enabled DSL flag makes it opt-in per target -- not every import needs the overhead.
The dry_running status follows the append-only rule for integer-backed enums, preserving existing data.
The records_will_change! call is the key to making StoreModel in-place mutations persist -- without it, ActiveRecord skips the attribute in the SQL UPDATE because its dirty tracking does not detect in-place changes on serialized objects.
The DryRunJob follows the same thin-job pattern as ParseJob and ImportJob: find, delegate, done.
The controller action and route mirror the existing member actions: POST triggers a job, redirect back to show.

Next up

We now have a full-featured, tested, and safe import engine. In part 15, we wrap up the series: publishing the gem to RubyGems, writing a proper CHANGELOG, choosing a versioning strategy, and reflecting on what worked, what we would do differently, and what DataPorter looks like from the outside.

This is part 14 of the series "Building DataPorter - A Data Import Engine for Rails". Previous: Testing a Rails Engine with RSpec | Next: Publishing the Gem & Retrospective

GitHub: SerylLns/data_porter | RubyGems: data_porter

Building a Rails Engine #13 -- How to test a mountable Rails engine without a dummy app

Seryl Lns — Tue, 24 Mar 2026 13:30:00 +0000

Testing a Rails Engine with RSpec

How to test a mountable Rails engine without a dummy app -- just an in-memory SQLite database and 60 lines of spec_helper.

Context

This is part 13 of the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 12, we extended the Source layer to support JSON files and API endpoints.

We have been writing specs throughout the series, but we never stepped back to explain how the test suite works. A Rails engine is not a Rails app. There is no config/database.yml. There is no spec/rails_helper.rb generated by rspec-rails. There is no schema to load, no test database to create, no ApplicationController to inherit from. Every piece of infrastructure that a typical Rails project gives you for free -- the database connection, the load paths, the base controller class -- has to be set up manually in a gem's test suite.

In this article, we look at the full testing strategy: the spec_helper that bootstraps a minimal Rails environment, the patterns we use for each layer (models, controllers, components, generators, JavaScript), and the TDD workflow that shaped the series.

The problem

A Rails engine gem ships as a library. It has no bin/rails, no config/application.rb, no running server. When you run rspec inside the gem, Ruby loads your spec_helper.rb and your spec files -- and that is it. If your code depends on ActiveRecord, you need a database. If it depends on ActionController, you need a controller base class. If it depends on load paths for app/models or app/controllers, you need to set those up yourself.

The conventional solution is a "dummy app" -- a minimal Rails application inside spec/dummy/ that boots the full Rails stack, mounts the engine, and provides the infrastructure specs need. This works, but it comes with overhead: you maintain a separate Gemfile.lock, a database config, an application class, route files, and all the boilerplate of a Rails app that exists only for tests. When the engine is small and focused, that overhead feels disproportionate.

DataPorter takes a different approach: no dummy app. The spec_helper.rb bootstraps just enough Rails to make the specs work, and nothing more. No application class. No route loading. No middleware stack. Just the frameworks we actually use -- ActiveRecord, ActiveJob, ActionController, ActionCable -- configured at the minimum viable level.

What we are building

Here is the spec_helper that powers the entire test suite:

# spec/spec_helper.rb
require "rails"
require "active_record"
require "active_job"
require "action_controller"
require "action_cable"
require "data_porter"

ActiveRecord::Base.establish_connection(
  adapter: "sqlite3",
  database: ":memory:"
)

ActiveRecord::Schema.define do
  create_table :data_porter_imports, force: true do |t|
    t.string  :target_key,  null: false, default: ""
    t.string  :source_type, null: false, default: "csv"
    t.integer :status,      null: false, default: 0
    t.text    :records
    t.text    :report
    t.text    :config

    t.string  :user_type
    t.integer :user_id

    t.timestamps
  end
end

%w[models jobs].each do |dir|
  $LOAD_PATH.unshift File.expand_path("../app/#{dir}", __dir__)
end
require "data_porter/data_import"
require "data_porter/parse_job"
require "data_porter/import_job"

# Stub for controller inheritance in test context
class ApplicationController < ActionController::Base; end unless defined?(ApplicationController)

$LOAD_PATH.unshift File.expand_path("../app/controllers", __dir__)
require "data_porter/imports_controller"

$LOAD_PATH.unshift File.expand_path("../app/channels", __dir__)
require "data_porter/import_channel"

$LOAD_PATH.unshift File.expand_path("../lib", __dir__)

RSpec.configure do |config|
  config.example_status_persistence_file_path = ".rspec_status"
  config.disable_monkey_patching!

  config.expect_with :rspec do |c|
    c.syntax = :expect
  end
end

Sixty lines of setup that replace a 200-file dummy app. Every line exists because a spec somewhere needs it.

Implementation

Step 1 -- The in-memory database

The first problem is the database. DataPorter's DataImport model is an ActiveRecord class that needs a real table with real columns. In a Rails app, you would run rails db:create db:migrate. In a gem, there is no migration runner. We need to create the schema from scratch every time the specs run.

SQLite's :memory: database solves this perfectly. Look at the first block in the spec_helper above: establish_connection creates a database that lives entirely in memory -- no file on disk, no cleanup needed, no state leaking between test runs. The Schema.define block creates the table immediately. By the time the first spec runs, the database exists with the right schema.

The schema here mirrors the migration that the install generator creates for host apps, but it is defined inline rather than loaded from a migration file. This is a deliberate duplication: the spec_helper schema is the test contract, not the migration itself. If the migration changes and we forget to update the spec_helper, a spec will fail -- which is exactly the signal we want.

The text columns for records, report, and config are where StoreModel does its work. In production with PostgreSQL, these would be jsonb columns. In the test suite with SQLite, they are text columns -- and StoreModel handles the serialization transparently in both cases. This is one of the advantages of StoreModel over raw JSONB: the serialization adapter abstracts the column type difference.

Step 2 -- Load paths and the ApplicationController stub

A Rails engine's code lives in app/models, app/controllers, app/jobs, and app/channels. In a running Rails app, the autoloader resolves these paths. In a gem's test suite, there is no autoloader. We add the directories to $LOAD_PATH manually and require each file explicitly (see the middle section of the spec_helper above). Without this, require "data_porter/data_import" would fail because Ruby does not know to look in app/models.

The controller layer is trickier. ImportsController inherits from the host app's ApplicationController (or whatever DataPorter.configuration.parent_controller is set to). In a host Rails app, that class already exists. In the gem's test suite, it does not. We stub it:

class ApplicationController < ActionController::Base; end unless defined?(ApplicationController)

The unless defined? guard is important. If something else in the test environment has already defined ApplicationController (which happens in some CI setups), we do not want to redefine it. The stub gives us just enough to verify that ImportsController is loadable, inherits from the right parent, and defines the expected methods. It does not give us routing or request processing -- but we do not need those for the kind of controller testing we do.

Step 3 -- Testing models and StoreModel objects

The DataImport model uses StoreModel for its records, report, and config attributes. The model specs test validations, enum behavior, and persistence:

# spec/data_porter/data_import_spec.rb
RSpec.describe DataPorter::DataImport, type: :model do
  let(:target_class) do
    Class.new(DataPorter::Target) do
      label "Guests"
      model_name "Guest"
      icon "fas fa-users"
    end
  end

  before do
    DataPorter::Registry.clear
    DataPorter::Registry.register(:guests, target_class)
  end

  describe "validations" do
    it "requires target_key" do
      import = described_class.new(source_type: "csv")

      expect(import).not_to be_valid
      expect(import.errors[:target_key]).to include("can't be blank")
    end
  end

  describe "persistence" do
    it "saves and reloads with records" do
      import = described_class.create!(
        target_key: "guests",
        source_type: "csv",
        user_type: "User",
        user_id: 1
      )
      record = DataPorter::StoreModels::ImportRecord.new(
        line_number: 1,
        data: { name: "Alice" }
      )
      import.update!(records: [record])

      reloaded = described_class.find(import.id)

      expect(reloaded.records.first.line_number).to eq(1)
      expect(reloaded.records.first.data).to eq({ "name" => "Alice" })
    end
  end
end

Two things stand out. First, every spec that depends on a target class creates an anonymous class with Class.new(DataPorter::Target) and registers it in the before block. This avoids polluting the global namespace with named test classes and ensures each spec controls its own target definition. The Registry.clear in before prevents target registrations from one spec leaking into another.

Second, the persistence spec exercises the full StoreModel round-trip: create a DataImport, add ImportRecord objects to the records attribute, save, reload from the database, and verify the data survived serialization. This is critical because StoreModel serializes to JSON under the hood, and the round-trip through SQLite's text column can surface subtle issues -- symbol keys becoming string keys, for instance, which is exactly what { "name" => "Alice" } (not { name: "Alice" }) is testing.

The ImportRecord StoreModel type has no table -- it lives inside a DataImport's records array. Testing it requires no database at all: instantiate with a hash, call methods, assert on state. The key spec verifies that #attributes returns symbolized, compacted data ({ name: "Alice" } not { "name" => "Alice", "email" => nil }), because the target's persist method depends on this contract.

Step 4 -- Testing controllers structurally

In a full Rails app, you would test controllers with request specs: send a POST, check the response status, verify the redirect. In a gem without a dummy app, there is no router, no middleware stack, and no request processing. We cannot send HTTP requests because there is nowhere to send them.

Instead, we test controllers structurally -- verifying the class hierarchy, the callback registrations, and the method signatures:

# spec/data_porter/imports_controller_spec.rb
RSpec.describe DataPorter::ImportsController do
  describe "inheritance" do
    it "inherits from the configured parent controller" do
      expect(described_class.superclass.name).to eq(
        DataPorter.configuration.parent_controller
      )
    end
  end

  describe "before_actions" do
    it "registers set_import callback" do
      callbacks = described_class._process_action_callbacks.select do |c|
        c.filter == :set_import
      end

      expect(callbacks).not_to be_empty
    end
  end

  describe "action methods" do
    it "defines index, new, create, show, parse, confirm, and cancel" do
      actions = %i[index new create show parse confirm cancel]
      actions.each do |action|
        expect(described_class.instance_method(action)).to be_a(UnboundMethod)
      end
    end
  end

  describe "private methods" do
    it "defines set_import" do
      expect(described_class.private_instance_methods).to include(:set_import)
    end

    it "defines import_params" do
      expect(described_class.private_instance_methods).to include(:import_params)
    end
  end
end

This approach tests four things that matter at the gem level: (1) the controller inherits from the right parent class, which proves the dynamic inheritance mechanism works; (2) the before_action callbacks are registered, which proves the controller's lifecycle is set up correctly; (3) every expected action method exists as an instance method; (4) the private helper methods exist and are private.

What this does not test is the actual behavior of those actions -- what happens when you call create with certain params, or whether show renders the right component. Those are integration concerns that belong in the host app's test suite, or in a separate integration test suite with a dummy app. The gem-level specs verify the contract -- the controller exists, has the right shape, and plugs into the right place. The host app tests verify the behavior.

The _process_action_callbacks API is an internal Rails method that returns the registered callbacks. It is not a public API, but it is stable across Rails versions and it is the only way to verify callback registration without actually processing a request. Using it here is a pragmatic choice -- the alternative is a full request spec, which requires a dummy app.

Step 5 -- Testing Phlex components

Phlex components are the easiest layer to test in a gem context because they have zero Rails dependencies. A Phlex component is a Ruby object with a call method that returns an HTML string. No view context, no request, no controller:

# spec/data_porter/components/preview_table_spec.rb
RSpec.describe DataPorter::Components::PreviewTable do
  def render(component)
    component.call
  end

  let(:target_class) do
    Class.new(DataPorter::Target) do
      label "Guests"
      model_name "Guest"
      columns do
        column :first_name, type: :string, required: true
        column :last_name, type: :string
      end
    end
  end

  let(:record) do
    DataPorter::StoreModels::ImportRecord.new(
      line_number: 1, status: "complete",
      data: { "first_name" => "Alice", "last_name" => "Smith" }
    )
  end

  it "renders column headers from target" do
    html = render(described_class.new(columns: target_class._columns, records: [record]))

    expect(html).to include("First name")
    expect(html).to include("Last name")
  end
end

The render helper is just component.call. The assertions are string-based (include("Alice")) rather than DOM-based (have_css("td", text: "Alice")) -- no Capybara, no Nokogiri. String matching is sufficient for verifying content in component output, and it keeps the test dependencies minimal.

Step 6 -- Testing generators

Generator specs follow the same structural pattern as controllers: verify inheritance, source root, and method definitions. We covered the full walkthrough in part 11 -- the key insight is that we test the generator class, not its filesystem output, because running rails generate would require a dummy app. This catches the most common bugs: misspelled method names, wrong source roots, or missing base classes.

Step 7 -- Testing JavaScript from Ruby specs

DataPorter ships a Stimulus controller in app/javascript/data_porter/progress_controller.js. In a typical frontend project, you would test this with Jest or Vitest, running the JavaScript in a Node.js environment. But DataPorter is a Ruby gem. Adding a JavaScript test runner means adding package.json, node_modules, and a separate CI step. For a single 30-line Stimulus controller, that is a lot of overhead.

Instead, we test the JavaScript file as a text artifact from Ruby:

# spec/data_porter/progress_controller_js_spec.rb
RSpec.describe "progress_controller.js" do
  let(:content) { File.read(js_path) }

  it "imports Stimulus Controller" do
    expect(content).to include('import { Controller } from "@hotwired/stimulus"')
  end

  it "subscribes to ImportChannel on connect" do
    expect(content).to include("DataPorter::ImportChannel")
  end

  it "updates progress bar width and text" do
    expect(content).to include("this.barTarget.style.width")
  end
end

Read the file as a string, assert it contains the expected patterns. No JavaScript execution, no Node.js toolchain. This catches a specific class of bugs: someone renames the ActionCable channel in Ruby but forgets to update the JS subscription string, or removes a Stimulus target from the HTML but leaves the declaration in the controller. These are integration seams where Ruby and JavaScript must agree, and string matching catches disagreements.

Is this a substitute for real JS testing? No. But for a single 30-line Stimulus controller, it provides meaningful coverage without adding package.json and node_modules to the project.

Step 8 -- The anonymous target class pattern

A pattern that runs through nearly every spec file is the anonymous target class:

let(:target_class) do
  Class.new(DataPorter::Target) do
    label "Guests"
    model_name "Guest"
    icon "fas fa-users"
    sources :csv

    columns do
      column :first_name, type: :string, required: true
      column :last_name, type: :string
      column :email, type: :email
    end

    csv_mapping do
      map "First Name" => :first_name
      map "Last Name" => :last_name
      map "Email" => :email
    end

    define_method(:persist) do |record, **|
      records << record.data
    end
  end
end

This creates a new class that inherits from DataPorter::Target without giving it a constant name. Each spec defines exactly the target it needs -- the Orchestrator spec defines one with persist, the CSV spec defines one with csv_mapping, the PreviewTable spec defines one with just columns. No spec depends on a shared target fixture that might change.

The define_method(:persist) pattern (instead of def persist) is required because anonymous class blocks use Class.new with a block, and def inside that block defines a method on the class -- but if the method needs to capture a local variable (like records for the collection array), define_method with a closure is the only way. This is a Ruby metaprogramming detail that comes up often when testing DSL-heavy code.

The companion pattern is registry cleanup:

before do
  DataPorter::Registry.clear
  DataPorter::Registry.register(:guests, target_class)
end

after { DataPorter::Registry.clear }

Every spec that registers a target clears the registry before and after. Without this, target registrations accumulate across specs, leading to flaky tests where the order of execution matters. The clear method on the Registry exists primarily for this testing use case.

Decisions & tradeoffs

Decision	We chose	Over	Because
Test infrastructure	Inline spec_helper with in-memory SQLite	Dummy Rails app in `spec/dummy/`	Less maintenance overhead, faster boot time, no separate Gemfile or config to maintain
Controller testing	Structural tests (inheritance, callbacks, method existence)	Request specs with a dummy app	Verifies the contract without requiring routing, middleware, or a full request cycle
Component testing	`component.call` with string assertions	Capybara or Nokogiri DOM assertions	Minimal dependencies; string matching is sufficient for verifying content in component output
Generator testing	Class-level structural tests	Running the generator against a temp directory	Catches wiring bugs without requiring a writable Rails app filesystem
JavaScript testing	Ruby string matching on file content	Jest/Vitest with a Node.js test runner	Avoids adding a JS toolchain for a single 30-line file; tests the Ruby/JS integration contract
Target fixtures	Anonymous classes per spec with `Class.new`	Shared named target classes	Each spec controls its own target definition; no coupling between specs
Registry isolation	`clear` before and after each spec that registers targets	Global shared state	Prevents registration leakage and ordering-dependent test failures
StoreModel testing	Direct instantiation without database	Round-trip through ActiveRecord	StoreModel objects are plain Ruby objects; database round-trips are tested separately on the DataImport model

The TDD workflow

In a Rails app, you can spike code, hit refresh, and see if it works. In a gem, there is no browser -- the spec suite is the feedback loop. Every feature in this series followed the same cycle: write a spec that describes the behavior, watch it fail, implement the minimum code to make it pass. The anonymous target classes, the structural controller tests, the JS string matching -- all of these patterns emerged from necessity, not theory. When you have no dummy app and no browser, you find the simplest way to verify each layer's contract, and that simplicity turns out to be an advantage.

Recap

No dummy app: the spec_helper bootstraps an in-memory SQLite database, sets up load paths, and stubs ApplicationController -- just enough Rails to test a full engine.
StoreModel objects are tested as plain Ruby objects, with database round-trip tests on the parent ActiveRecord model.
Controllers are tested structurally -- inheritance, callbacks, and method existence -- rather than with request specs, because the gem has no router.
Phlex components are tested by calling .call and asserting against the HTML string, with no view context or browser.
Generators are tested by verifying class hierarchy, source root, and method definitions -- the wiring, not the filesystem output.
JavaScript files are tested as text artifacts from Ruby, verifying the contract between the JS and the rest of the engine.
Anonymous target classes with Class.new keep each spec self-contained, and Registry.clear prevents state leakage.

Next up

Now that we have a tested, fully-featured engine, there is one more safety net to add before imports touch the database. In part 14, we build Dry Run -- a validation mode that wraps the import in a transaction, rolls it back, and enriches records with the database-level errors that the preview phase cannot catch. Preview validates the data; dry run validates the persistence.

This is part 13 of the series "Building DataPorter - A Data Import Engine for Rails". Previous: Adding JSON & API Sources | Next: Dry Run: Validate Before You Persist

GitHub: SerylLns/data_porter | RubyGems: data_porter

Building a Rails Engine #12 -- Beyond CSV: JSON and API Sources

Seryl Lns — Thu, 19 Mar 2026 13:30:00 +0000

Beyond CSV: JSON and API Sources

CSV is the king of data import -- but in the real world, data also arrives as JSON from a file, or directly from a third-party API. Here is how DataPorter extends its source architecture to absorb these new formats without breaking anything.

Context

This is part 12 of the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 11, we built the install and target generators so adopting the gem requires a single command.

Until now, DataPorter only reads CSV. That covers a lot of cases, but the real world is more varied. If every new format requires rearchitecting the engine, something went wrong. The Sources::Base abstraction we established in part 6 is about to prove its worth.

Interactive Walkthrough →

DataPorter Sources Architecture — Explaina

Interactive explorer showing how one fetch contract powers CSV, JSON, and API sources — pick a source type and see the code.

explaina.app

Why multiple sources?

An import engine that only speaks CSV forces users to convert their data before importing. That is unnecessary friction. Your hotel partner sends a nightly JSON export of guest reservations. Your CRM exposes a REST API with paginated contacts. A front-end form lets operators paste raw JSON into a text field. Each of these is a valid data source, and none of them is a CSV.

By supporting JSON and API natively, we cover all three without asking users to convert anything first. The key point: every source must respect the same contract -- a fetch method that returns an array of hashes with symbol keys. The rest of the pipeline (validation, transformation, persistence) does not change.

The JSON source

The JSON source must handle three ways to receive content: direct injection (for tests or programmatic use), raw JSON stored in the import configuration, and download from an ActiveStorage attachment.

# lib/data_porter/sources/json.rb
module DataPorter
  module Sources
    class Json < Base
      def initialize(data_import, content: nil)
        super(data_import)
        @content = content
      end

      def fetch
        parsed = ::JSON.parse(json_content)
        records = extract_records(parsed)

        Array(records).map do |hash|
          hash.transform_keys { |k| k.parameterize(separator: "_").to_sym }
        end
      end

      private

      def json_content
        @content || config_raw_json || download_file
      end

      def config_raw_json
        config = @data_import.config
        config["raw_json"] if config.is_a?(Hash)
      end

      def download_file
        @data_import.file.download
      end

      def extract_records(parsed)
        root = @target_class._json_root
        return parsed unless root

        parsed.dig(*root.split("."))
      end
    end
  end
end

Three things stand out.

The json_content cascade. The method tries three sources in order: content injected into the constructor, the raw_json key in the import configuration, and finally the ActiveStorage file. This cascade allows great flexibility without explicit configuration -- the right path is chosen automatically based on what is available.

json_root for nested paths. Real-world APIs and JSON files often wrap data in a structure: {"data": {"guests": [...]}}. Rather than forcing the user to flatten their JSON, we give them a DSL method in the Target:

class GuestsTarget < DataPorter::Target
  label "Guests"
  model_name "Guest"
  json_root "data.guests"

  columns do
    column :name, type: :string
  end
end

The extract_records method uses dig by splitting the path on dots. "data.guests" becomes parsed.dig("data", "guests"). Simple, readable, and supports any level of nesting.

Key normalization. As with the CSV source, every key is transformed via parameterize(separator: "_").to_sym. "First Name" becomes :first_name. This guarantees the rest of the pipeline always receives keys in the same format, regardless of the source format.

The JSON source covers file-based and programmatic use cases. But sometimes the data lives behind an HTTP endpoint, and the engine needs to go fetch it.

The API source

The API source fetches data from an HTTP endpoint. It must support static and dynamic endpoints, fixed and lazily-generated headers, and data extraction from a response key.

# lib/data_porter/sources/api.rb
module DataPorter
  module Sources
    class Api < Base
      def fetch
        api = @target_class._api_config
        response = perform_request(api)
        parsed = ::JSON.parse(response.body)
        records = extract_records(parsed, api)

        Array(records).map do |hash|
          hash.transform_keys { |k| k.parameterize(separator: "_").to_sym }
        end
      end

      private

      def perform_request(api)
        url = resolve_endpoint(api)
        headers = resolve_headers(api)
        uri = URI(url)

        Net::HTTP.start(uri.host, uri.port, use_ssl: uri.scheme == "https") do |http|
          request = Net::HTTP::Get.new(uri)
          headers.each { |k, v| request[k] = v }
          http.request(request)
        end
      end

      def resolve_endpoint(api)
        params = @data_import.config.symbolize_keys
        api.endpoint.is_a?(Proc) ? api.endpoint.call(params) : api.endpoint
      end

      def resolve_headers(api)
        api.headers.is_a?(Proc) ? api.headers.call : (api.headers || {})
      end

      def extract_records(parsed, api)
        root = api.response_root
        root ? parsed[root.to_s] : parsed
      end
    end
  end
end

The core logic lives in resolve_endpoint and resolve_headers. Each accepts either a static value or a lambda. This opens two usage modes:

# Static endpoint, fixed headers
api_config do
  endpoint "https://api.example.com/stays"
  headers({ "Authorization" => "Bearer abc123" })
  response_root :stays
end

# Dynamic endpoint, lazily-generated headers
api_config do
  endpoint ->(params) { "https://api.example.com/items?id=#{params[:item_id]}" }
  headers(-> { { "Authorization" => "Bearer #{Token.current}" } })
end

For the endpoint lambda, parameters come from @data_import.config.symbolize_keys. The user passes config: { item_id: "42" } when creating the import, and the lambda receives those parameters to build the URL. For headers, the lambda is called with no argument -- it retrieves the token from wherever it lives (environment variable, model, external service).

response_root works like json_root from the JSON source, but simpler: it extracts a single key from the response hash. response_root :stays on a response {"stays": [...]} returns the array directly. If no response_root is defined, the entire response is used.

The ApiConfig DSL pattern

The API configuration uses a dedicated DSL object rather than plain attr_accessor:

# lib/data_porter/dsl/api_config.rb
module DataPorter
  module DSL
    class ApiConfig
      def endpoint(value = nil)
        return @endpoint if value.nil?

        @endpoint = value
      end

      def headers(value = nil)
        return @headers if value.nil?

        @headers = value
      end

      def response_root(value = nil)
        return @response_root if value.nil?

        @response_root = value
      end
    end
  end
end

Each method plays a dual role: called with an argument, it acts as a setter; called without, it acts as a getter. In the Target, api_config creates an ApiConfig instance and executes the block via instance_eval -- the same DSL object + instance_eval pattern we used for the columns block. A classic Ruby idiom that gives clean syntax while keeping the implementation testable as a plain PORO.

Dispatch via Sources.resolve

Adding new sources does not modify any existing code. The Sources module maintains a simple registry:

# lib/data_porter/sources.rb
module DataPorter
  module Sources
    REGISTRY = {
      api: Api,
      csv: Csv,
      json: Json
    }.freeze

    def self.resolve(type)
      REGISTRY.fetch(type.to_sym) { raise Error, "Unknown source type: #{type}" }
    end
  end
end

The Orchestrator calls Sources.resolve(import.source_type) and receives the right class. It then instantiates the source and calls fetch. Neither the Orchestrator nor the controllers know which source type is being used -- it is the source_type stored in the import that decides. Adding an XML or Parquet source would require: a class inheriting from Base, one entry in the REGISTRY, and nothing else.

The TDD approach

Both sources were built test-first. The JSON source specs cover each path through the cascade -- direct injection, json_root extraction, and raw_json fallback:

it "parses JSON array content" do
  json = '[{"first_name": "Alice", "last_name": "Smith"}]'
  source = described_class.new(import, content: json)

  expect(source.fetch.first[:first_name]).to eq("Alice")
end

it "extracts records from a nested path" do
  json = '{"data": {"guests": [{"name": "Alice"}, {"name": "Bob"}]}}'
  source = described_class.new(import_with_root, content: json)

  expect(source.fetch.size).to eq(2)
end

The API source specs stub Net::HTTP.start and test the same axes: static vs lambda endpoint, header resolution, and response_root extraction. We are not testing that Net::HTTP works -- we are testing that our code correctly composes the URL, headers, and extracts the right data from the response.

Decisions & tradeoffs

Decision	We chose	Over	Because
HTTP client	`Net::HTTP` (stdlib)	Faraday, HTTParty	Zero extra dependency; sufficient for simple GETs
Dynamic endpoint	Lambda receiving `params`	String with interpolation	The lambda allows any logic (conditions, service calls) without string eval
Dynamic headers	Lambda with no argument	Callback with context	Headers often come from a global service (ENV, token store), not the import context
JSON cascade	`content` > `raw_json` > `file`	Mandatory argument	Maximum flexibility; each use case finds its natural path
Key normalization	`parameterize` + `to_sym`	Explicit mapping	Consistent with the CSV source; the downstream pipeline always receives the same format

Recap

The JSON source supports three input modes (injection, config raw_json, file) via a cascade of fallbacks, and uses json_root to navigate nested structures.
The API source dynamically resolves endpoints and headers through a static/lambda dual system, and extracts data via response_root.
The ApiConfig DSL uses a getter/setter pattern without attr_reader, evaluated in an instance_eval block for natural syntax.
Sources.resolve dispatches to the right class via a frozen registry -- adding a source is a two-line operation.
Tests cover every path through every source without touching the network, thanks to content injection and HTTP stubbing.

Next up

JSON and API sources complete the trio of supported formats. But we have not yet talked about the engine's overall testing strategy -- how to test a Rails engine without a full host application, how to organize specs between unit and integration tests, how to mock ActiveStorage and ActionCable. In part 13, we dive into testing a Rails engine with RSpec and the patterns that keep the suite fast and reliable.

This is part 12 of the series "Building DataPorter - A Data Import Engine for Rails". Previous: Generators: Install & Target Scaffolding | Next: Testing a Rails Engine with RSpec

Turn Messy WhatsApp Messages Into Structured JSON-Reliably

Seryl Lns — Tue, 17 Mar 2026 18:00:00 +0000

Designing Deterministic Outputs From Unstructured Messages

Your users send messy, unstructured text. Your backend expects clean, validated JSON. Bridging that gap is the hard part — and most teams get it wrong.

This article walks through how to turn a raw WhatsApp message into a structured payload your application can safely act on.

The Problem

A real message from a hotel guest:

hey can u move bk-4521 to fri plzz?? also ned more towls in rm 312 thx

Typos. Abbreviations. Two requests in one message. No punctuation.

Your PMS expects:

{
  "booking_ref": "BK-4521",
  "new_date": "2026-03-14",
  "room": "312",
  "items": ["extra towels"]
}

How do you get from A to B — reliably?

Step 1: Define Your Schema

Before touching any LLM, define what your backend expects. This is your contract.

{
  "intent": "string (enum)",
  "actions": [
    {
      "type": "string (action identifier)",
      "payload": {
        "field_1": "string | number | null",
        "field_2": "string | number | null"
      }
    }
  ],
  "confidence": "number (0-1)",
  "suggested_reply": "string | null"
}

Key principles:

Finite set of intents — your agent only handles what you define
Typed fields — each field has an expected type
Nullable — missing data is null, not hallucinated
Confidence score — so your backend can decide when to auto-act vs. ask for confirmation

Step 2: Extract, Don't Converse

The classic mistake is building a multi-turn flow to gather missing fields. Instead, extract everything from the first message in a single LLM call.

The messy message:

hey can u move bk-4521 to fri plzz?? also ned more towls in rm 312 thx

Gets processed into:

{
  "intent": "modify_booking",
  "confidence": 0.93,
  "actions": [
    {
      "type": "reservation.booking.reschedule",
      "payload": {
        "booking_ref": "BK-4521",
        "new_date": "2026-03-14"
      }
    },
    {
      "type": "housekeeping.task.create",
      "payload": {
        "room": "312",
        "items": "extra towels"
      }
    }
  ],
  "suggested_reply": "Done! Booking BK-4521 moved to Friday. Extra towels are on the way to room 312."
}

One message → two actions. No follow-up questions needed.

Step 3: Validate Before Acting

Never trust raw LLM output blindly. Validate the payload before executing:

# Your webhook handler
def handle_agent_webhook(payload)
  output = payload["data"]["output"]

  # Check confidence threshold
  return if output["confidence"] < 0.8

  output["actions"].each do |action|
    case action["type"]
    when "booking.reschedule"
      # Validate booking exists
      booking = Booking.find_by(ref: action.dig("payload", "booking_ref"))
      next unless booking

      booking.reschedule!(new_date: action.dig("payload", "new_date"))

    when "housekeeping.task.create"
      HousekeepingTask.create!(
        room: action.dig("payload", "room"),
        items: action.dig("payload", "items")
      )
    end
  end

  # Send the suggested reply back via WhatsApp
  if output["suggested_reply"]
    send_whatsapp_message(
      session_id: payload["data"]["session_id"],
      text: output["suggested_reply"]
    )
  end
end

Your backend stays in control. The LLM proposes, your code decides.

Step 4: Handle the Edges

What about messages that don't match any intent?

"What's the weather like in Paris tomorrow?"

Output:

{
  "intent": "system.noop.skip",
  "confidence": 0.9,
  "actions": [],
  "suggested_reply": null,
  "analysis_summary": "The message is unrelated to hotel services."
}

No action taken. No hallucinated response. No made-up answer about Paris weather. The system knows what it doesn't handle and stays silent.

This is the difference between a deterministic system and a chatbot that tries to answer everything. A chatbot would hallucinate a weather forecast. A message-driven system returns noop and moves on.

The Full Pipeline

Raw message
  ↓
Pre-filter (spam, noise → free, no LLM)
  ↓
Router (classify → which agent handles this?)
  ↓
Agent (LLM → structured payload with schema)
  ↓
Webhook (HMAC-signed → your backend)
  ↓
Validate (confidence check, field validation)
  ↓
Execute (create task, update booking, send reply)

Explore the full pipeline in detail →

Each layer adds reliability:

Pre-filter removes noise before LLM cost
Router ensures the right agent handles each message type
Schema constrains LLM output to valid shapes
Confidence lets you set auto-action thresholds
Webhooks decouple processing from action

What You Get

Metric	Typical chatbot	Message-driven
Round-trips per request	3-5	1
Latency	5-15s (multi-turn)	1-2s
State management	Complex	None
Accuracy on intent	~70%	95%+ (on our hotel dataset)
Handles typos/slang	Poorly	Well
Backend integration	Custom per-flow	Standard webhook

This pipeline works because it treats messages as data extraction problems, not conversations. Define the shape, let the LLM fill it, validate before acting. If you want to skip building it yourself:

Try It Yourself

WhatsRB Cloud handles this entire pipeline — from raw WhatsApp message to structured webhook. Define your intents, set your schema, and let the platform do the extraction.

Beta opens March 31, 2026.

Join the waitlist → | Read the API docs →

No chatbot framework. No dialog trees. Just clean data from messy messages.

Built with Ruby, Rails, and the belief that most messages don't need a conversation — they need an action.

How We Route WhatsApp Messages to N Agents With a Single LLM Call

Seryl Lns — Tue, 17 Mar 2026 16:58:28 +0000

Most teams building AI-powered messaging systems make the same mistake: they run every inbound message through every agent. Got 5 agents? That's 5 LLM calls per message. Your users send "👍" and you just burned $0.003 classifying a thumbs-up five times.

We needed a better approach. Here's the pipeline we built — and why each layer exists.

The Naive Approach (And Why It Hurts)

The obvious architecture:

Message arrives
  → Agent 1: "Is this for me?" (LLM call)
  → Agent 2: "Is this for me?" (LLM call)
  → Agent 3: "Is this for me?" (LLM call)
  → Agent 4: "Is this for me?" (LLM call)
  → Agent 5: "Is this for me?" (LLM call)

5 LLM calls. 5× the latency. 5× the cost. And 4 of them will say "nah, not for me."

Now imagine your hotel WhatsApp gets 500 messages/day. That's 2,500 LLM calls just for routing. Most of them are "ok", "👍", "thx", and emoji reactions.

You're paying OpenAI to classify thumbs-ups.

The Pipeline: Free Before Paid

Our philosophy is simple: filter what you can for free, before you spend tokens.

Let's walk through each layer.

Layer 1: Trailing-Edge Debounce

People don't send one message. They send three:

14:02:01  "I need room service"
14:02:04  "room 204"
14:02:07  "before 2pm if possible"

Without debounce, you'd process "I need room service" and miss the room number. The agent would hallucinate it or ask a follow-up question. Neither is great.

We use a trailing-edge debounce: each inbound message schedules a delayed job (say, 3 seconds). When the job fires, it checks if newer messages arrived for the same conversation. If yes, it bails — a fresher job is already in the queue.

The result: "I need room service" + "room 204" + "before 2pm" get batched into one pipeline run.

No Redis streams. No WebSocket aggregation. Just a background job with a delay and a freshness check. Dead simple.

The debounce window is configurable per agent. Too short and you split messages. Too long and response feels slow. We default to 3 seconds — short enough to feel instant, long enough to catch the "oh wait, one more thing" follow-up.

Layer 2: PreFilter — Kill Noise for Free

Before spending a single token, we filter the obvious junk with plain regex:

Empty messages — webhook noise, media-only messages with no text
Emoji-only — "👍", "😂😂😂", "🙏" (yes, we handle emoji ZWJ sequences)
Ultra-short — "ok", "ty", "k" (below a configurable minimum length)

That's it. Three rules. Zero LLM cost. Takes less than a millisecond.

In production, this filters ~30-40% of all inbound messages before the LLM even wakes up. On a busy account, that's hundreds of dollars saved per month.

We thought about adding language detection, spam scoring, sentiment analysis... We didn't. Three regex rules catch enough, and every rule we add is a rule we need to maintain. The LLM router handles the harder cases anyway.

Layer 3: The Single-LLM Router

This is the core trick. Instead of asking each agent "is this for you?", we ask one cheap model to classify AND route in a single call.

The approach: build a tool-calling schema where the LLM must return:

actionable (boolean) — is this a real request or just noise?
reason — why it classified this way (for audit)
agent_ids — which agents should handle this message

The key insight is constraining the tool schema dynamically. The agent_ids parameter uses an enum that's generated from the user's actual agent IDs:

{
  "agent_ids": {
    "type": "array",
    "items": {
      "type": "string",
      "enum": ["uuid-concierge", "uuid-housekeeping", "uuid-billing"]
    }
  }
}

The LLM literally cannot hallucinate an agent ID — it can only pick from the list. And the system prompt is also dynamic, listing each agent's name and a truncated description of what it handles.

One call. A tiny model (we use gpt-4.1-nano). Costs about ~$0.00015 per message. Routes to 1, 2, or all agents at once.

On a real account with ~400 messages/day, the router costs about $2/month. Without it, running 3 agents on every message would cost ~$15/month in classification alone.

Fail-Open: Never Lose a Message

What happens when the LLM provider is down?

We fail open. If the router call fails (after fallback — more on that in our next article), we route to ALL agents. Each agent's own LLM call will decide if the message is relevant.

You might burn a few extra tokens during an outage, but you'll never silently drop a customer message. This is a deliberate trade-off: false positives over false negatives. A customer whose message gets processed by an extra agent won't notice. A customer whose message disappears into the void will leave a 1-star review.

Layer 4: Per-Agent Filters

After routing, each agent gets one more chance to reject — based on config, not LLM:

Trigger mode — only trigger on text messages, skip images/audio/stickers
Min message length — ignore very short messages
Ignore patterns — regex patterns to skip (e.g., /^(ok|thx|merci)$/i)
Rate limit — max N messages/hour per conversation (prevents loops)
Excluded numbers — blocklist specific phone numbers (internal, test devices)

Two design decisions worth highlighting:

1. User config can only make filters stricter, never weaker.

If our default minimum message length is 3 characters, a user can set it to 5 but not to 0. They can't set trigger_on: "all" if the default is "text_only". Opinionated? Yes. But it prevents a class of "my agent is running on every message and I'm out of quota" support tickets.

2. User-defined regex runs with a hard timeout.

Because users will write (.+)+$ and wonder why their server is melting. We timeout after 100ms and skip the broken pattern. ReDoS protection is not optional when you accept user-defined regex.

The Audit Trail

Every routing decision gets logged with:

Outcome — routed, skip_rules, skip_router, no_agent_matched, error_fallback
Per-agent decisions — which agents were routed, which were filtered, and why
Token usage — how many tokens the router consumed

When a user asks "why didn't my agent process this message?", we can answer with data, not guesswork. And on our end, we can spot patterns — if error_fallback spikes, something is wrong upstream.

Interactive Demo →

whatsrb.com

The Numbers

On a real account (hotel, ~400 messages/day, 3 agents):

Layer	Messages filtered	Cost
PreFilter (regex)	~35%	$0
Router (not actionable)	~25%	~$0.06/day
Per-agent filter	~5%	$0
Actually processed	~35%	Agent LLM cost

65% of messages never reach an agent. The first and last filters are free. The router in the middle costs pennies.

What We'd Do Differently

The debounce window is the hardest tuning problem. There's no perfect value. We expose it as a per-agent config and let users decide. Most never change the default.
Fail-open is the right default, but it needs monitoring. Without tracking error_fallback outcomes, you'd never know your router has been down for 3 hours and every message is hitting every agent. Alerting on this is critical.
Don't over-engineer the PreFilter. We almost built a spam classifier. We're glad we didn't. The simplest possible rules give you 80% of the value for 0% of the maintenance.

Original Article

This Is What We Built

This pipeline powers WhatsRB Cloud — a platform that turns WhatsApp messages into structured, actionable webhooks. You define agents, set routing rules, and receive clean JSON payloads.

If you're building something similar on WhatsApp, we've done the hard part so you don't have to.

whatsrb.com

Get started → | API docs →

Built with Ruby, Rails, and a deep respect for not wasting tokens on thumbs-ups.

Building a Rails Engine #11 -- Generators: Install & Target Scaffolding

Seryl Lns — Tue, 17 Mar 2026 13:30:00 +0000

Generators: Install & Target Scaffolding

A great gem installs in one command. A great engine scaffolds new import types from the command line. Here is how to build Rails generators that bootstrap everything -- migration, initializer, routes, and per-target files -- so adopters never have to wire anything by hand.

Context

This is part 11 of the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 10, we built the ImportsController, wired up engine routes, and solved the dynamic parent controller inheritance problem.

At this point the engine is feature-complete: targets, sources, the orchestrator, real-time progress, a Phlex UI, and controllers all work together. But onboarding a new host app still requires manually creating a migration, writing an initializer, mounting the engine in routes, and knowing the exact Target DSL to define an import type. That is too many steps for someone evaluating the gem for the first time. We need generators that collapse all of that into a single rails generate command.

Interactive Walkthrough →

DataPorter Generators Walkthrough — Explaina

Interactive step-by-step demo showing how Rails generators eliminate manual setup — from 4 error-prone steps to a single command.

explaina.app

The problem

Installing a Rails engine by hand means at least four discrete steps: copy a migration, create an initializer with sane defaults, add a route mount, and create the directory where import targets will live. Miss any one of these and the engine will not work -- but the error messages will not tell you which step you forgot. A missing migration produces an ActiveRecord::StatementInvalid; a missing route mount means a NoMethodError when the engine tries to resolve its URL helpers; a missing initializer silently uses defaults that may not match your app.

On top of that, every time a developer wants to add a new import type, they need to remember the Target DSL: which class to inherit from, which methods to define, how to declare columns. That is cognitive overhead that belongs in a generator, not in someone's memory.

What we are building

With generators, it looks like this.

Two generators. The first bootstraps the entire engine into a host app:

$ rails generate data_porter:install
      create  db/migrate/20260206120000_create_data_porter_imports.rb
      create  config/initializers/data_porter.rb
      create  app/importers
       route  mount DataPorter::Engine, at: "/imports"

The second scaffolds a new target with parsed column definitions:

$ rails generate data_porter:target guests first_name:string:required email:email last_name:string
      create  app/importers/guests_target.rb

One command to install, one command per import type. No manual file creation, no DSL memorization.

Implementation

Step 1 -- The install generator

The install generator inherits from Rails::Generators::Base and mixes in ActiveRecord::Generators::Migration for timestamped migration support. Each public method in the generator becomes a step that Rails executes in definition order:

# lib/generators/data_porter/install/install_generator.rb
module DataPorter
  module Generators
    class InstallGenerator < Rails::Generators::Base
      include ActiveRecord::Generators::Migration

      source_root File.expand_path("templates", __dir__)

      def copy_migration
        migration_template(
          "create_data_porter_imports.rb.erb",
          "db/migrate/create_data_porter_imports.rb"
        )
      end

      def copy_initializer
        template("initializer.rb", "config/initializers/data_porter.rb")
      end

      def create_importers_directory
        empty_directory("app/importers")
      end

      def mount_engine
        route 'mount DataPorter::Engine, at: "/imports"'
      end
    end
  end
end

Four methods, four steps. Let us walk through each one.

copy_migration uses migration_template instead of the plain template method. The difference matters: migration_template adds a timestamp prefix to the filename, turning a static template into a proper Rails migration. It also raises if the migration already exists -- no accidental duplicates. The template itself is an ERB file that interpolates the current ActiveRecord migration version:

# lib/generators/data_porter/install/templates/create_data_porter_imports.rb.erb
class CreateDataPorterImports < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
  def change
    create_table :data_porter_imports do |t|
      t.string  :target_key,  null: false
      t.string  :source_type, null: false, default: "csv"
      t.integer :status,      null: false, default: 0
      t.jsonb   :records,     null: false, default: []
      t.jsonb   :report,      null: false, default: {}
      t.jsonb   :config,      null: false, default: {}

      t.references :user, polymorphic: true, null: false

      t.timestamps
    end

    add_index :data_porter_imports, :status
    add_index :data_porter_imports, :target_key
  end
end

The <%= ActiveRecord::Migration.current_version %> call means the generated migration always matches the host app's Rails version -- a Rails 7.1 app gets Migration[7.1], a Rails 7.2 app gets Migration[7.2]. No hardcoding, no compatibility issues.

copy_initializer generates a commented-out configuration file. Every option is present but disabled, so developers can see what is available without digging through source code:

# lib/generators/data_porter/install/templates/initializer.rb
DataPorter.configure do |config|
  # Parent controller for the engine's controllers to inherit from.
  # Defaults to ActionController::Base. Set to "ApplicationController"
  # to inherit authentication, layouts, and helpers from your app.
  # config.parent_controller = "ApplicationController"

  # Context builder: inject business data into targets.
  # Receives the DataImport record.
  # config.context_builder = ->(data_import) {
  #   { user: data_import.user }
  # }

  # Enabled source types.
  # config.enabled_sources = %i[csv json xlsx api]
end

The full initializer includes additional options for queue name, storage service, cable prefix, and preview limits -- each with the same self-documenting pattern. The key design choice: every option documents itself with a comment that explains what it controls, not just what it is. When someone opens this file for the first time, they should understand the engine's configuration surface without reading the README.

create_importers_directory calls empty_directory, which creates app/importers and silently skips if it already exists. This is where host apps will place their target files.

mount_engine uses the built-in route helper, which injects the mount line into config/routes.rb. The default mount point is /imports, but because it is a standard route mount, developers can change the path or wrap it in constraints after generation.

The engine is installed. Now let us make it just as easy to add new import types.

Step 2 -- The target generator

The target generator inherits from Rails::Generators::NamedBase instead of Base. This gives us the name argument for free -- Rails automatically parses the first argument as the target name and provides helper methods like class_name, file_name, and singular_name:

# lib/generators/data_porter/target/target_generator.rb
module DataPorter
  module Generators
    class TargetGenerator < Rails::Generators::NamedBase
      source_root File.expand_path("templates", __dir__)

      argument :columns, type: :array, default: [], banner: "name:type[:required]"

      def create_target_file
        template("target.rb.tt", "app/importers/#{file_name}_target.rb")
      end

      private

      def target_class_name
        "#{class_name}Target"
      end

      def model_name
        class_name.singularize
      end

      def target_label
        class_name.titleize
      end

      def parsed_columns
        columns.map { |col| parse_column(col) }
      end

      def parse_column(definition)
        parts = definition.split(":")
        {
          name: parts[0],
          type: parts[1] || "string",
          required: parts[2] == "required"
        }
      end
    end
  end
end

The columns argument uses type: :array with a default: [], which means you can generate a bare target with no columns and fill them in later, or you can specify everything up front. The banner string "name:type[:required]" tells rails generate data_porter:target --help exactly what format columns expect.

The parse_column method splits each column definition on colons. first_name:string:required becomes { name: "first_name", type: "string", required: true }. email:email becomes { name: "email", type: "email", required: false }. If you omit the type entirely, it defaults to "string". This mirrors the familiar rails generate model syntax but adapts it to our column DSL.

The naming helpers derive everything from the single name argument. Pass guests and you get: target_class_name returns "GuestsTarget", model_name returns "Guest" (singularized, because the target usually maps to one ActiveRecord model), and target_label returns "Guests" (titleized, for the UI).

Step 3 -- The target template

The template uses Thor's .tt format (which processes ERB tags using the generator's binding) to produce a complete, working target file:

# lib/generators/data_porter/target/templates/target.rb.tt
class <%= target_class_name %> < DataPorter::Target
  label "<%= target_label %>"
  model_name "<%= model_name %>"
  icon "fas fa-file-import"
  sources :csv
<% if parsed_columns.any? %>

  columns do
<% parsed_columns.each do |col| -%>
    column :<%= col[:name] %>, type: :<%= col[:type] %><%= ", required: true" if col[:required] %>
<% end -%>
  end
<% end %>

  def persist(record, context:)
    # <%= model_name %>.create!(record.attributes)
  end
end

Running rails generate data_porter:target guests first_name:string:required email:email last_name:string produces:

# app/importers/guests_target.rb
class GuestsTarget < DataPorter::Target
  label "Guests"
  model_name "Guest"
  icon "fas fa-file-import"
  sources :csv

  columns do
    column :first_name, type: :string, required: true
    column :email, type: :email
    column :last_name, type: :string
  end

  def persist(record, context:)
    # Guest.create!(record.attributes)
  end
end

Two details worth noting. First, the persist method is generated with a commented-out implementation line. This is intentional: we want the developer to think about what persistence means for their domain rather than blindly accepting a default. Maybe they need find_or_create_by, maybe they need to call a service object, maybe they need to update existing records. The commented hint shows the simplest path while making it clear this is the one method they must customize.

Second, the columns block is conditionally rendered. If you run rails generate data_porter:target guests with no columns, you get a clean skeleton without an empty columns do; end block. You can add columns later as you figure out your CSV structure.

Decisions & tradeoffs

Decision	We chose	Over	Because
Generator base class	`NamedBase` for target, `Base` for install	Both using `Base`	`NamedBase` gives us `class_name`, `file_name`, and argument parsing for free; install does not need a name argument
Migration template format	ERB (`.rb.erb`) with `ActiveRecord::Migration.current_version`	Hardcoded migration version	The generated migration automatically matches the host app's Rails version
Column parsing syntax	`name:type[:required]` colon-separated	A flag-based approach (`--columns name:string --required name`)	Matches the familiar `rails generate model` convention; less typing, easier to remember
Initializer style	All options commented out with explanations	Only showing uncommented defaults	Developers discover every configuration option on first read; uncommenting is easier than looking up what is available
Target output directory	`app/importers/`	`app/data_porter/targets/` or `app/models/concerns/`	Short, clear, conventional; mirrors how apps organize service objects in `app/services/`
Persist method	Commented-out hint (`# Guest.create!(...)`)	Working default implementation	Forces the developer to make an intentional choice about their persistence strategy

Testing it

Generator testing is unusual -- you are testing file creation, not object behavior. The install generator specs are structural (right inheritance, right source root, right methods), so let us focus on the more interesting target generator specs that test column parsing and naming derivation:

# spec/data_porter/generators/target_generator_spec.rb
RSpec.describe DataPorter::Generators::TargetGenerator do
  describe "column parsing" do
    let(:generator) { described_class.new(["guests", "first_name:string:required", "email:email"]) }

    it "parses column definitions" do
      columns = generator.send(:parsed_columns)

      expect(columns.size).to eq(2)
      expect(columns[0]).to eq({ name: "first_name", type: "string", required: true })
      expect(columns[1]).to eq({ name: "email", type: "email", required: false })
    end
  end

  describe "naming" do
    let(:generator) { described_class.new(["guests"]) }

    it "derives the class name" do
      expect(generator.send(:target_class_name)).to eq("GuestsTarget")
    end

    it "derives the model name" do
      expect(generator.send(:model_name)).to eq("Guest")
    end

    it "derives the label" do
      expect(generator.send(:target_label)).to eq("Guests")
    end
  end
end

The column parsing spec confirms that first_name:string:required correctly splits into a hash with required: true, while email:email (no third segment) defaults to required: false. The naming specs verify that guests as input produces GuestsTarget for the class, Guest for the model, and Guests for the label -- the singularization and titleization that the template depends on.

Notice that the specs instantiate the generator directly with described_class.new(["guests", ...]) rather than invoking it through the Rails generator runner. This keeps the tests fast and focused: we are testing the logic, not the file I/O.

Recap

The install generator bootstraps the entire engine in one command: migration, initializer, importers directory, and route mount -- four steps that previously required reading the README and manually creating files.
The target generator scaffolds a new import type with parsed column definitions, producing a ready-to-customize target file that follows the DSL conventions established in earlier parts of the series.
The migration template uses ERB with ActiveRecord::Migration.current_version to match the host app's Rails version automatically, avoiding hardcoded version numbers.
The column parsing syntax (name:type[:required]) mirrors rails generate model conventions, keeping the learning curve flat for Rails developers.
The initializer template documents every configuration option with comments, making the engine's surface area discoverable without external documentation.

Next up

The engine can now parse CSV files, but real-world data does not always arrive in a spreadsheet. In part 12, we extend the Source layer with JSON and API sources -- a JSON source that accepts files or raw text, and an API source that fetches data from HTTP endpoints with injectable parameters and authentication headers. The source abstraction we designed in part 6 is about to prove its worth.

This is part 11 of the series "Building DataPorter - A Data Import Engine for Rails". Previous: Controllers & Routing in a Rails Engine | Next: Adding JSON & API Sources

Building a Rails Engine #10 -- Controllers & Routing in a Rails Engine

Seryl Lns — Wed, 11 Mar 2026 17:36:48 +0000

Controllers & Routing in a Rails Engine

Engine controllers are tricky -- they need to inherit from the host app, define routes that do not collide with anything else, and stay thin while coordinating an entire import workflow. Here is the clean way.

Context

This is part 10 of the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 9, we built the UI layer with Phlex components and scoped Tailwind styles, giving users a visual interface for previewing and managing imports.

But a UI needs a backend to drive it. We need controller actions that create imports, kick off background jobs, let users confirm or cancel, and route everything under a clean, isolated namespace. In this article, we build the ImportsController, wire up engine routes with member actions, and solve one of the trickiest problems in engine development: making the controller inherit from whichever parent class the host app wants.

The parent controller pattern

When you build a Rails engine, your controllers need to inherit from something. The obvious choice is ActionController::Base -- it works, no dependencies. But then the host app's authentication, layouts, and error handling do not flow through. The user would need to separately configure auth for every engine route.

The other extreme is inheriting from ApplicationController. That gives you auth for free, but crashes when the host app uses authorization gems (Pundit, CanCanCan) that enforce after_action verification callbacks on ApplicationController.

The solution is dynamic inheritance. The engine defaults to ActionController::Base -- safe, no implicit dependencies -- but the host app can swap it to any controller in one line:

# lib/data_porter/configuration.rb
class Configuration
  attr_accessor :parent_controller,
                :queue_name,
                :storage_service,
                :cable_channel_prefix,
                :context_builder,
                :preview_limit,
                :enabled_sources,
                :scope

  def initialize
    @parent_controller = "ActionController::Base"
    @queue_name = :default
    @storage_service = :local
    @cable_channel_prefix = "data_porter"
    @context_builder = nil
    @preview_limit = 500
    @enabled_sources = %i[csv json api]
    @scope = nil
  end
end

The controller then uses constantize at class definition time to resolve the string into a class:

# app/controllers/data_porter/imports_controller.rb
module DataPorter
  class ImportsController < DataPorter.configuration.parent_controller.constantize

This single line is doing a lot. When Ruby loads the controller file, it evaluates the superclass expression. DataPorter.configuration.parent_controller returns a string like "ActionController::Base" or "Admin::BaseController", and .constantize turns it into the actual class. The result is that ImportsController inherits from whatever the host app configured -- picking up its before_actions, authentication, layouts, and helper methods automatically.

Out of the box, the engine works without authentication (inheriting from ActionController::Base). A host app that wants authentication to flow through just sets the parent controller in the initializer:

# config/initializers/data_porter.rb
DataPorter.configure do |config|
  config.parent_controller = "ApplicationController"
  # or a dedicated admin controller:
  # config.parent_controller = "Admin::BaseController"
end

No monkey-patching, no middleware, no route constraints. The engine's controller is an admin controller.

The tradeoff is that this happens once, at load time. If you change the configuration after the controller class has been loaded, the inheritance does not update. In practice this is not an issue -- initializers run before controllers are loaded in production, and in development Rails reloads everything per-request anyway. But it is worth knowing: the parent controller is resolved eagerly, not lazily.

Engine routes

DataPorter draws its routes inside the engine's own router, completely isolated from the host app:

# config/routes.rb
DataPorter::Engine.routes.draw do
  resources :imports, only: %i[index new create show] do
    member do
      post :parse
      post :confirm
      post :cancel
    end
  end
end

The host app mounts the engine at whatever path it wants:

# Host app: config/routes.rb
mount DataPorter::Engine, at: "/data_porter"

This produces the following route table:

HTTP method	Path	Action	Purpose
GET	`/data_porter/imports`	`index`	List all imports
GET	`/data_porter/imports/new`	`new`	New import form
POST	`/data_porter/imports`	`create`	Create and start parsing
GET	`/data_porter/imports/:id`	`show`	Preview/status page
POST	`/data_porter/imports/:id/parse`	`parse`	Re-parse an import
POST	`/data_porter/imports/:id/confirm`	`confirm`	Confirm and start importing
POST	`/data_porter/imports/:id/cancel`	`cancel`	Cancel an import

Notice what is not here: edit, update, destroy. An import is not a CRUD resource in the traditional sense. You create it, you preview it, you confirm or cancel it. There is no "edit an import" workflow -- if the data is wrong, you cancel and create a new one. The routes reflect this by using only: to restrict the standard REST actions and adding three custom member routes for the state-transition actions.

The member routes are all post, not patch or put. These are command actions that trigger side effects (enqueue a job, change status), not updates to a resource's attributes. POST is semantically correct here and plays well with Turbo, which sends POST requests from button_to helpers by default.

Because isolate_namespace is set on the engine, all routes are automatically scoped. Inside the controller, you use import_path(@import) and imports_path -- no prefix needed. The engine's router handles the namespace. This also means the host app's own imports_path (if it has one) is completely separate.

Each action explained

Here is the full controller, then we will walk through each action:

# app/controllers/data_porter/imports_controller.rb
module DataPorter
  class ImportsController < DataPorter.configuration.parent_controller.constantize
    before_action :set_import, only: %i[show parse confirm cancel]

    def index
      @imports = DataPorter::DataImport.order(created_at: :desc)
    end

    def new
      @import = DataPorter::DataImport.new
      @targets = DataPorter::Registry.available
    end

    def create
      @import = DataPorter::DataImport.new(import_params)
      @import.user = current_user if respond_to?(:current_user, true)
      @import.status = :pending

      if @import.save
        DataPorter::ParseJob.perform_later(@import.id)
        redirect_to import_path(@import)
      else
        @targets = DataPorter::Registry.available
        render :new
      end
    end

    def show
      @target = @import.target_class
      @records = @import.records
      @grouped = @records.group_by(&:status)
    end

    def parse
      @import.update!(status: :pending)
      DataPorter::ParseJob.perform_later(@import.id)
      redirect_to import_path(@import)
    end

    def confirm
      @import.update!(status: :pending)
      DataPorter::ImportJob.perform_later(@import.id)
      redirect_to import_path(@import)
    end

    def cancel
      @import.update!(status: :failed)
      redirect_to imports_path
    end

    private

    def set_import
      @import = DataPorter::DataImport.find(params[:id])
    end

    def import_params
      params.require(:data_import).permit(:target_key, :source_type, :file, config: {})
    end
  end
end

index -- Lists all imports, newest first. No pagination yet (that is a future enhancement), but the query is simple enough that the view layer can handle it. This is the dashboard view.

new -- Builds a blank DataImport and loads the available targets from the Registry. The view uses these to render a dropdown of import types. This is where the Target DSL pays off: each registered target provides its own label and icon, and the controller just passes the list through.

create -- The most interesting action. It builds the import from strong params, optionally assigns the current user (using respond_to?(:current_user, true) to avoid crashing if the host app has no authentication), sets the initial status, and saves. On success, it immediately enqueues a ParseJob and redirects to the show page where the user will see real-time progress. On failure, it reloads the targets and re-renders the form.

The respond_to?(:current_user, true) check deserves attention. The second argument (true) includes private methods in the check. Some authentication gems define current_user as a private helper. By passing true, we catch both public and private definitions. If the host app has no authentication at all, the check returns false and the user association is simply not set. The import still works -- user is a polymorphic optional association.

show -- Loads the target class, the records, and groups them by status. The view uses this grouping to show separate sections for complete, partial, and missing records. This is the preview page when the import is in previewing status, and the results page when it is completed.

parse -- Re-parses an existing import. This resets the status to pending and enqueues a new ParseJob. Useful when the user realizes the CSV mapping was wrong and re-uploads a file, or when a previous parse failed and they want to retry.

confirm -- The user has reviewed the preview and decided to proceed. This resets the status to pending, enqueues an ImportJob, and redirects back to the show page, where ActionCable will push real-time progress updates (as we built in part 8). The status reset follows the same pattern as parse -- setting pending before enqueuing ensures the UI shows the right state immediately.

cancel -- Marks the import as failed and redirects to the index. No job is enqueued, no further processing happens. The import is preserved in the database for auditing but is effectively dead.

Strong params

The import_params method is deliberately minimal:

def import_params
  params.require(:data_import).permit(:target_key, :source_type, :file, config: {})
end

Four permitted fields:

target_key -- A string like "user_import" that maps to a registered target. The Target DSL and Registry validate this downstream.
source_type -- A string like "csv", "json", or "api". Determines which Source class handles the data.
file -- An ActiveStorage attachment. The uploaded CSV, JSON, or other file.
config -- A hash for source-specific configuration (API endpoints, headers, JSON root paths). The config: {} syntax permits any keys inside the hash, which is intentional: different source types need different config keys, and we validate them at the Source level rather than the controller level. This is safe because config is a JSONB column -- the values are data, not model attributes. There is no mass assignment risk.

Notably absent: status, records, report, user_id, user_type. These are all set internally -- status by the controller and orchestrator, records and report by the ParseJob, and user by the controller's current_user logic. Strong params prevents any of these from being injected through the form.

Decisions & tradeoffs

Decision	We chose	Over	Because
Parent controller	Dynamic inheritance via `constantize`, defaulting to `ActionController::Base`	Hardcoded `ApplicationController`	Works out of the box without auth dependencies; host app opts in explicitly; avoids crashes from authorization gem callbacks
Route design	`only:` with custom member actions	Full `resources` CRUD	Imports are not a traditional CRUD resource; explicit routes match the actual workflow (create, preview, confirm/cancel)
Member action verbs	All POST	PATCH for parse, DELETE for cancel	These are command actions with side effects, not attribute updates; POST is semantically correct and works naturally with Turbo's `button_to`
User assignment	`respond_to?(:current_user, true)` guard	Requiring authentication	The engine must work with and without authentication; optional user association keeps it flexible
Config params	`config: {}` (open hash)	Explicitly listing every config key	Different source types need different config keys; validation happens at the Source level where context exists
No edit/update/destroy	Omitted from routes entirely	Including them "just in case"	An import is an immutable workflow; if it is wrong, you cancel and start over; fewer routes means fewer security surfaces

Testing it

Testing engine controllers without a full Rails app or a dummy app is the interesting challenge here. The approach we use: stub just enough of the Rails stack to load the controller class, then test its structure directly.

The spec helper handles the setup. It loads the minimal Rails dependencies, sets up an in-memory SQLite database, stubs ApplicationController, and manually requires the controller:

# spec/spec_helper.rb (relevant excerpt)
require "rails"
require "action_controller"

# Stub for controller inheritance in test context
class ApplicationController < ActionController::Base; end unless defined?(ApplicationController)

$LOAD_PATH.unshift File.expand_path("../app/controllers", __dir__)
require "data_porter/imports_controller"

The unless defined?(ApplicationController) guard is important -- it prevents double-definition errors if the host app's ApplicationController has already been loaded (which happens in integration test suites).

With that foundation, the controller spec tests structure rather than HTTP behavior:

# spec/data_porter/imports_controller_spec.rb
RSpec.describe DataPorter::ImportsController do
  let(:target_class) do
    Class.new(DataPorter::Target) do
      label "Test Import"
      icon "fas fa-test"
      model_name "TestModel"

      columns do
        column :name, type: :string, required: true
      end
    end
  end

  before do
    DataPorter::Registry.clear
    DataPorter::Registry.register(:test_import, target_class)
  end

  after { DataPorter::Registry.clear }

  describe "inheritance" do
    it "inherits from the configured parent controller" do
      expect(described_class.superclass.name).to eq(DataPorter.configuration.parent_controller)
    end
  end

  describe "before_actions" do
    it "registers set_import callback" do
      callbacks = described_class._process_action_callbacks.select do |c|
        c.filter == :set_import
      end

      expect(callbacks).not_to be_empty
    end
  end

  describe "action methods" do
    it "defines index, new, create, show, parse, confirm, and cancel" do
      actions = %i[index new create show parse confirm cancel]
      actions.each do |action|
        expect(described_class.instance_method(action)).to be_a(UnboundMethod)
      end
    end
  end

  describe "private methods" do
    it "defines set_import and import_params" do
      expect(described_class.private_instance_methods).to include(:set_import, :import_params)
    end
  end
end

The tests verify structure rather than HTTP behavior: inheritance matches the config, callbacks are registered, action methods exist. Callback registration is tested via _process_action_callbacks -- a Rails internal API, but stable and far simpler than making real HTTP requests just to check that a before_action is wired up.

The routes get their own spec that redraws the engine routes and inspects the route set:

# spec/data_porter/routes_spec.rb
RSpec.describe "DataPorter routes" do
  before(:all) do
    DataPorter::Engine.routes.draw do
      resources :imports, only: %i[index new create show] do
        member do
          post :parse
          post :confirm
          post :cancel
        end
      end
    end
  end

  it "defines import resource routes" do
    routes = DataPorter::Engine.routes
    route_set = routes.routes.map { |r| [r.defaults[:action], r.path.spec.to_s] }

    expect(route_set).to include(["index", "/imports(.:format)"])
    expect(route_set).to include(["new", "/imports/new(.:format)"])
    expect(route_set).to include(["create", "/imports(.:format)"])
    expect(route_set).to include(["show", "/imports/:id(.:format)"])
  end

  it "defines member routes for parse, confirm, and cancel" do
    routes = DataPorter::Engine.routes
    route_set = routes.routes.map { |r| [r.defaults[:action], r.path.spec.to_s] }

    expect(route_set).to include(["parse", "/imports/:id/parse(.:format)"])
    expect(route_set).to include(["confirm", "/imports/:id/confirm(.:format)"])
    expect(route_set).to include(["cancel", "/imports/:id/cancel(.:format)"])
  end
end

The before(:all) block redraws the routes inside the test environment. This is necessary because the engine's routes file is loaded differently in test versus production. By explicitly drawing the routes, we guarantee the test is exercising the exact route definitions we ship, not whatever routes happened to be loaded by a host app. The tradeoff: if you change the routes file, you must update the spec too -- but that is true of any test that asserts on structure. The specs then map each route to an [action, path] pair and verify all seven are present with the correct paths.

This style of testing -- structural rather than behavioral -- may feel unusual. But for an engine, it is the right level of abstraction. Full request specs belong in the host app's integration suite, where they can exercise the real authentication stack, the real database, and the real views. The engine's specs verify that the pieces are correctly defined and wired together.

Recap

The dynamic parent controller pattern uses constantize to resolve the configured controller name at class-load time, allowing the engine to inherit authentication, layouts, and helpers from whatever base class the host app chooses.
Engine routes are drawn inside DataPorter::Engine.routes.draw, keeping them isolated from the host app. The host mounts the engine at any path it wants.
Seven actions map to a clean import workflow: list, create, preview, re-parse, confirm, cancel. No edit or destroy -- imports are immutable workflows.
Strong params permit only the fields the user should control (target_key, source_type, file, config), while status, records, and user are set internally.
Controller specs test structure (inheritance, callbacks, method existence) rather than HTTP behavior, avoiding the need for a full dummy app.

Next up

The controller and routes give us a working web interface, but setting up DataPorter in a host app still requires creating a migration, writing an initializer, mounting routes, and knowing the right directory for target classes. In part 11, we build install and target generators -- rails generate data_porter:install that handles the entire setup in one command, and rails generate data_porter:target that scaffolds a new import type with columns pre-filled from the command line.

This is part 10 of the series "Building DataPorter - A Data Import Engine for Rails". Previous: Building the UI with Phlex & Tailwind | Next: Generators: Install & Target Scaffolding

Building a Rails Engine #9 -- Building the UI with Phlex & Tailwind

Seryl Lns — Tue, 10 Mar 2026 13:00:00 +0000

Building the UI with Phlex & Tailwind

How to build a full component library for a Rails engine using Phlex, scope all CSS with a dp- prefix so styles never leak into the host app, and generate preview tables dynamically from the Target DSL.

Context

This is part 9 of the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 8, we wired up ActionCable and Stimulus so users get real-time progress updates as imports run in the background.

We now have a working engine: targets define imports, sources parse files, the Orchestrator coordinates the workflow, and ActionCable pushes updates to the browser. But the browser has nothing to render. Every status, every preview row, every progress bar needs a component. In this article, we build the entire view layer -- six Phlex components (plus a shared Base class) that turn DataPorter's internal data structures into HTML the user can actually see.

Why Phlex (not ERB or ViewComponent)

A Rails engine's view layer needs to ship inside the gem. ERB templates in engines are fragile -- they depend on the host app's layout, helpers, and asset pipeline. ViewComponent still relies on template resolution. Phlex takes a different approach: the template is the Ruby class. A component is a plain object with a view_template method that uses a Ruby DSL to emit HTML. No template files, no implicit dependencies. Every component lives in lib/, ships with the gem, and renders anywhere -- in a controller, in a test, in a Turbo Stream. For a gem, this is ideal.

Implementation

Step 1 -- The Base component

Every DataPorter component inherits from a shared base class that extends Phlex::HTML:

# lib/data_porter/components/base.rb
require "phlex"

module DataPorter
  module Components
    class Base < Phlex::HTML
    end
  end
end

This is intentionally minimal. The base class exists as a single point of inheritance so we can add shared behavior later -- a common CSS helper, a default wrapper, an HTML safety policy -- without touching every component. Right now it does nothing but establish the hierarchy. That is fine. The value is in the seam, not the code.

The module barrel file requires all components in order:

# lib/data_porter/components.rb
require_relative "components/base"
require_relative "components/status_badge"
require_relative "components/summary_cards"
require_relative "components/preview_table"
require_relative "components/progress_bar"
require_relative "components/results_summary"
require_relative "components/failure_alert"

module DataPorter
  module Components
  end
end

Step 2 -- StatusBadge: the simplest component

The StatusBadge renders a single <span> with the import's current status. It demonstrates the pattern every component follows:

# lib/data_porter/components/status_badge.rb
module DataPorter
  module Components
    class StatusBadge < Base
      def initialize(status:)
        super()
        @status = status.to_s
      end

      def view_template
        span(class: "dp-badge dp-badge--#{@status}") { @status.capitalize }
      end
    end
  end
end

The constructor takes a keyword argument and calls super() -- Phlex requires this. The view_template method emits a span with two CSS classes: a base class dp-badge for shared badge styles, and a modifier class dp-badge--#{@status} for status-specific colors. The naming follows BEM conventions, but with the dp- prefix to scope everything to DataPorter.

Rendering it is a method call:

DataPorter::Components::StatusBadge.new(status: "completed").call
# => '<span class="dp-badge dp-badge--completed">Completed</span>'

No partial lookup, no view context, no render helpers. Just instantiate and call.

Step 3 -- SummaryCards: parsing report data into a dashboard

After the Orchestrator's parse! phase completes, the import has a Report object with counts for each record status. The SummaryCards component turns that into a row of four cards:

# lib/data_porter/components/summary_cards.rb
module DataPorter
  module Components
    class SummaryCards < Base
      def initialize(report:)
        super()
        @report = report
      end

      def view_template
        div(class: "dp-summary-cards") do
          card("dp-card--complete", @report.complete_count, "Ready")
          card("dp-card--partial", @report.partial_count, "Incomplete")
          card("dp-card--missing", @report.missing_count, "Missing")
          card("dp-card--duplicate", @report.duplicate_count, "Duplicates")
        end
      end

      private

      def card(css_class, count, label)
        div(class: "dp-card #{css_class}") do
          strong { count.to_s }
          plain " #{label}"
        end
      end
    end
  end
end

The private card method extracts the repeated pattern: a div with a modifier class, a bold count, and a label. Each card gets a status-specific class (dp-card--complete, dp-card--partial, etc.) so the host app's stylesheet -- or DataPorter's own default styles -- can color-code them. The plain helper emits raw text without wrapping it in an element, keeping the markup minimal. It is text-safe -- no HTML escaping issues, unlike raw.

This component takes a single report: argument and does not care where the report came from. In production it comes from data_import.report. In a test, you can pass a plain Report.new(complete_count: 10, ...). The component has no database dependency.

Step 4 -- PreviewTable: dynamic columns from the Target DSL

The PreviewTable is the most complex component and the one that justifies choosing Phlex. It builds an HTML table whose columns are defined dynamically by the Target class's column DSL -- the same DSL we built in part 5. A target with two columns produces a table with two data columns. A target with ten produces ten. No template changes needed.

# lib/data_porter/components/preview_table.rb
module DataPorter
  module Components
    class PreviewTable < Base
      def initialize(columns:, records:)
        super()
        @columns = columns
        @records = records
      end

      def view_template
        div(class: "dp-preview-table") do
          table(class: "dp-table") do
            render_header
            render_body
          end
        end
      end

      private

      def render_header
        thead do
          tr do
            th { "#" }
            th { "Status" }
            @columns.each { |col| th { col.label } }
            th { "Errors" }
          end
        end
      end

      def render_body
        tbody do
          @records.each { |record| render_row(record) }
        end
      end

      def render_row(record)
        tr(class: "dp-row--#{record.status}") do
          td { record.line_number.to_s }
          td { record.status }
          @columns.each { |col| td { record.data[col.name.to_s].to_s } }
          td(class: "dp-errors") { error_messages(record) }
        end
      end

      def error_messages(record)
        record.errors_list.map(&:message).join(", ")
      end
    end
  end
end

The header row always starts with a line number column and a status column, then iterates over the target's _columns to produce one <th> per declared column using col.label (the human-readable name), and ends with an errors column. The body does the same iteration for each record, pulling values from record.data by column name.

Each row gets a dp-row--#{record.status} class. A row with status complete gets dp-row--complete, a row with missing gets dp-row--missing. This lets the stylesheet highlight problem rows -- red background for missing data, yellow for partial, green for complete -- without any conditional logic in the component.

The key design choice here is that the PreviewTable takes columns: and records: as separate arguments rather than a DataImport object. This keeps it decoupled: the component does not need to know how to resolve a target or load records from the database. The controller (or whoever renders it) passes the data in, and the component turns it into HTML.

Step 5 -- ProgressBar: bridging Phlex and Stimulus

The ProgressBar component is where server-rendered Phlex meets client-side Stimulus. It renders the initial HTML with Stimulus data attributes, and the Stimulus controller (from part 8) takes over to animate the bar as ActionCable pushes updates:

# lib/data_porter/components/progress_bar.rb
module DataPorter
  module Components
    class ProgressBar < Base
      def initialize(import_id:)
        super()
        @import_id = import_id
      end

      def view_template
        div(class: "dp-progress", **stimulus_controller_attrs) do
          render_bar
        end
      end

      private

      def render_bar
        div(class: "dp-progress-bar",
            data_data_porter__progress_target: "bar",
            style: "width: 0%") do
          span(data_data_porter__progress_target: "text") { "0%" }
        end
      end

      def stimulus_controller_attrs
        {
          data_controller: "data-porter--progress",
          data_data_porter__progress_id_value: @import_id.to_s
        }
      end
    end
  end
end

The Stimulus naming convention for engines uses the data-porter--progress format -- double dash separating the namespace from the controller name. Phlex's HTML DSL converts Ruby keyword arguments to HTML attributes using a simple mapping:

Ruby keyword	HTML output
`data_controller`	`data-controller`
`data_data_porter__progress_target`	`data-data-porter--progress-target`

Single underscores become dashes. Double underscores (__) become double dashes (--), which is exactly what Stimulus expects for namespaced controllers.

The component renders the bar at width: 0%. The Stimulus controller subscribes to the ActionCable channel using the id_value and updates the width and text as progress events arrive. The Phlex component's only job is to emit the correct initial HTML with the right data attributes. It does not know about ActionCable or JavaScript.

Step 6 -- ResultsSummary and FailureAlert: post-import components

After the import completes, two components display the outcome. ResultsSummary shows the final counts:

# lib/data_porter/components/results_summary.rb
module DataPorter
  module Components
    class ResultsSummary < Base
      def initialize(report:)
        super()
        @report = report
      end

      def view_template
        div(class: "dp-results") do
          p { "Created: #{@report.imported_count}" }
          p { "Errors: #{@report.errored_count}" }
        end
      end
    end
  end
end

FailureAlert handles the catastrophic failure case -- when the Orchestrator catches an exception and transitions to failed:

# lib/data_porter/components/failure_alert.rb
module DataPorter
  module Components
    class FailureAlert < Base
      def initialize(report:)
        super()
        @report = report
      end

      def view_template
        div(class: "dp-alert dp-alert--danger") do
          @report.error_reports.each do |err|
            p { err.message }
          end
        end
      end
    end
  end
end

Both are intentionally simple. The ResultsSummary renders two paragraphs. The FailureAlert iterates over error reports and renders each message. There is no conditional logic, no branching, no fallback states. The controller decides which component to render based on the import's status. The components just render what they are given.

The `dp-` CSS prefix strategy

Every CSS class in the component library starts with dp-: dp-badge, dp-table, dp-progress, dp-alert, dp-card, dp-row, dp-errors, dp-results, dp-summary-cards, dp-preview-table. This is a manual namespacing strategy that solves a real problem: a Rails engine's styles must coexist with the host app's styles without collisions.

If DataPorter used generic class names like badge, table, or alert, they would conflict with Bootstrap, Tailwind UI, or whatever the host app uses. If we used CSS modules or Shadow DOM, we would add complexity and browser constraints that a server-rendered Rails app should not need.

The dp- prefix is the simplest approach that works. It is a convention, not a mechanism -- there is nothing stopping someone from using dp-badge in their own app. But it is unlikely, and it is obvious when it happens. The naming follows BEM-style modifiers (dp-badge--failed, dp-row--complete, dp-card--partial) so the structure is predictable.

DataPorter ships with a default stylesheet that targets these classes. Host apps can override any of them. Because every class is prefixed, a blanket override like .dp-badge { ... } will only affect DataPorter's badges without touching anything else on the page.

This strategy also plays well with Tailwind. If the host app uses Tailwind, they can style DataPorter's components with @apply inside a dp--prefixed selector, or they can use Tailwind's @layer to scope overrides. The prefix gives them a clean hook without requiring any configuration in DataPorter itself.

Decisions & tradeoffs

Decision	We chose	Over	Because
Component framework	Phlex	ERB partials, ViewComponent	Components are plain Ruby objects in `lib/`; no template resolution, no sidecar files, no asset pipeline dependency
CSS scoping	Manual `dp-` prefix on all classes	CSS modules, Shadow DOM, Tailwind prefix config	Simplest approach that prevents collisions; no build step, no browser constraints, works everywhere
Naming convention	BEM-style with `dp-` namespace (`dp-badge--failed`)	Flat class names, utility classes	Predictable structure; modifiers make status-specific styling obvious
PreviewTable inputs	Separate `columns:` and `records:` arguments	A single `data_import:` argument	Decouples the component from the database; testable with plain objects
ProgressBar strategy	Server-render initial HTML, let Stimulus animate	Render progress entirely client-side	The initial state (0%) is valid HTML; progressive enhancement means the page works even if JS fails to load
Component granularity	Seven small components	One large "import view" component	Each component is independently testable, replaceable, and composable
Base class	Empty `Base < Phlex::HTML`	Components inherit directly from `Phlex::HTML`	Provides a seam for future shared behavior without touching every component

Testing it

Phlex components are pure Ruby objects, which means testing them does not require a view context, a request, or a Rails controller. You instantiate the component and call call to get the HTML string:

# spec/data_porter/components/status_badge_spec.rb
RSpec.describe DataPorter::Components::StatusBadge do
  def render(component)
    component.call
  end

  it "renders a span with status text" do
    html = render(described_class.new(status: "completed"))
    expect(html).to include("Completed")
    expect(html).to include("dp-badge")
  end

  it "includes status-specific CSS class" do
    html = render(described_class.new(status: "failed"))
    expect(html).to include("dp-badge--failed")
  end

  it "renders all valid statuses" do
    %w[pending parsing previewing importing completed failed].each do |status|
      html = render(described_class.new(status: status))
      expect(html).to include(status.capitalize)
    end
  end
end

The pattern is the same for every component: define a local render helper that calls component.call, construct the component with the right arguments, and assert against the HTML string. No Capybara, no request specs, no browser.

The PreviewTable spec is the most interesting because it uses an anonymous Target class to generate columns. An anonymous class avoids polluting the test suite with named constants that could leak between specs:

# spec/data_porter/components/preview_table_spec.rb
RSpec.describe DataPorter::Components::PreviewTable do
  let(:target_class) do
    Class.new(DataPorter::Target) do
      label "Guests"
      model_name "Guest"

      columns do
        column :first_name, type: :string, required: true
        column :last_name, type: :string
      end
    end
  end

  let(:record) do
    DataPorter::StoreModels::ImportRecord.new(
      line_number: 1,
      status: "complete",
      data: { "first_name" => "Alice", "last_name" => "Smith" }
    )
  end

  it "renders column headers from target" do
    html = render(described_class.new(columns: target_class._columns, records: [record]))

    expect(html).to include("First name")
    expect(html).to include("Last name")
  end

  it "renders record data in rows" do
    html = render(described_class.new(columns: target_class._columns, records: [record]))

    expect(html).to include("Alice")
    expect(html).to include("Smith")
  end

  it "includes status-specific row class" do
    html = render(described_class.new(columns: target_class._columns, records: [record]))
    expect(html).to include("dp-row--complete")
  end
end

This test proves the full loop: the Target DSL declares columns, the PreviewTable reads those column definitions, and the rendered HTML contains the right headers and data. If someone adds a column to the target, the table grows automatically. No template change required.

The ProgressBar spec verifies the Stimulus wiring without needing JavaScript:

# spec/data_porter/components/progress_bar_spec.rb
RSpec.describe DataPorter::Components::ProgressBar do
  it "renders a progress bar with Stimulus data attributes" do
    html = render(described_class.new(import_id: 42))

    expect(html).to include("dp-progress")
    expect(html).to include("data-controller")
    expect(html).to include("data-porter--progress")
    expect(html).to include("42")
  end

  it "renders bar and text targets" do
    html = render(described_class.new(import_id: 1))

    expect(html).to include('data-data-porter--progress-target="bar"')
    expect(html).to include('data-data-porter--progress-target="text"')
  end
end

We do not test that the bar animates. We test that the HTML contract between Phlex and Stimulus is correct -- the right controller name, the right target names, the right value attribute. If those are present, the Stimulus controller (tested separately) will work.

Recap

Phlex lets us ship components as plain Ruby classes in lib/, with no template files, no view context, and no asset pipeline dependency -- ideal for a Rails engine gem.
The dp- prefix on every CSS class prevents style collisions with the host app, using the simplest possible approach: a naming convention.
The PreviewTable builds its columns dynamically from the Target DSL, so adding a column to a target automatically adds a column to the preview -- no template changes.
The ProgressBar bridges server-rendered Phlex and client-side Stimulus by emitting the correct data attributes at render time, then letting JavaScript handle animation.
Every component is independently testable by calling component.call and asserting against the returned HTML string -- no browser, no request context, no Capybara.

Next up

We have components, but nothing renders them yet. In part 10, we build the controllers and routing layer for the engine -- how ImportsController inherits from the host app's parent controller, how engine routes are namespaced, and how Turbo integration ties the components to the import workflow. The tricky part: making authentication flow from the host app into the engine without coupling to any specific auth library.

This is part 9 of the series "Building DataPorter - A Data Import Engine for Rails". Previous: Real-time Progress with ActionCable & Stimulus | Next: Controllers & Routing in a Rails Engine

Stop Building Chatbots. Start Building Message-Driven Systems

Seryl Lns — Mon, 09 Mar 2026 15:06:37 +0000

Most teams building on WhatsApp are building chatbots. Multi-turn conversations, intent trees, fallback loops — the whole nine yards.

But here's the thing: most of them don't actually need a chatbot.

They need a system that turns incoming messages into backend actions.

The Chatbot Trap

A hotel guest sends:

"Can I get extra towels in room 312?"

The typical chatbot approach:

Bot: Hi! How can I help you?
Guest: I need towels
Bot: Which room?
Guest: 312
Bot: How many towels?
Guest: ...just some towels
Bot: I didn't understand. How many towels would you like?

3 round-trips to extract 2 fields. The guest is annoyed. The developer is maintaining a conversation state machine. Everyone loses.

Messages Are Not Conversations

Here's the insight: most inbound messages already contain everything you need. You don't need to ask follow-up questions — you need to extract what's already there.

The shift is simple:

Message
  ↓
Analysis (single LLM call)
  ↓
Structured payload
  ↓
Backend action

One message in, one structured payload out. No conversation state. No dialog trees. No "I didn't understand."

See it in action →

What This Looks Like in Practice

The same message — "Can I get extra towels in room 312?" — processed as a message-driven system:

{
  "intent": "room_service_request",
  "confidence": 0.97,
  "actions": [
    {
      "type": "housekeeping.task.create",
      "payload": {
        "room": "312",
        "item": "extra towels"
      }
    }
  ],
  "suggested_reply": "Of course! Extra towels are on the way to room 312."
}

Your backend receives this via webhook. You create the task in your PMS. You send the reply. Done.

One message. One payload. One action.

The Architecture

Here's the full flow — from WhatsApp message to backend action:

WhatsApp (inbound message)
  ↓
Pre-filter (spam, noise — no LLM cost)
  ↓
Router (classify + route to the right agent)
  ↓
Agent (LLM call → structured output)
  ↓
Webhook (HMAC-signed payload to your app)
  ↓
Your backend (create task, update booking, send reply)

Each step is deterministic. The LLM is used once, for extraction — not for conversation. The output is a structured JSON payload your backend can trust.

Code, Not Config

Here's what triggering an agent looks like with the Ruby SDK (same way with TypeScript or Python SDK):

require "whatsrb"

client = WhatsRB::Client.new(api_key: "wrb_live_xxx")

run = client.agents.run(
  agent_id: "agt_abc123",
  input: "Cancel booking #BK-4521"
)

puts run.output
# => { "intent" => "cancel_booking", "actions" => [...] }

Or with a simple curl:

curl -X POST "https://api.whatsrb.com/v1/agents/agt_abc123/runs" \
  -H "Authorization: Bearer wrb_live_xxx" \
  -H "Content-Type: application/json" \
  -d '{"input":"Cancel booking #BK-4521"}'

You get back a structured payload. Every time. No conversation state to manage.

When Chatbots Make Sense (And When They Don't)

Use case	Chatbot?	Message-driven?
Customer FAQ with branching logic	Yes	No
"Book a table for 4 at 8pm"	No	Yes
"Cancel my order #12345"	No	Yes
"AC broken in room 204"	No	Yes
Guided product recommendation	Yes	No
"Where's my shipment TRK-789?"	No	Yes

If the message already contains the intent and the data — you don't need a chatbot. You need a message-to-action pipeline.

Real Results

With this approach:

No conversation state to maintain
Single LLM call per message (~1-2s latency)
Deterministic outputs your backend can trust
90%+ accuracy on intent + field extraction out of the box

This Is What We're Building

WhatsRB Cloud is a platform that turns WhatsApp messages into structured, actionable webhooks. No chatbot framework. No dialog trees. Just messages in, actions out.

We're opening the beta on March 31, 2026 to a limited group of early adopters.

Get early access →

Check out the API docs to see how it works under the hood.

Follow me on X for updates
GitHub

Built with Ruby, Rails, and a healthy distrust of chatbots.

Building a Rails Engine #8 — Real-time Progress with ActionCable & Stimulus

Seryl Lns — Thu, 05 Mar 2026 13:00:00 +0000

Real-time Progress with ActionCable & Stimulus

How to push live progress updates from a background import job to the browser using a Broadcaster service, an ActionCable channel, and a Stimulus controller -- so users never stare at a dead spinner again.

Context

This is part 8 of the series where we build DataPorter, a mountable Rails engine for data import workflows. In part 7, we built the Orchestrator -- the class that coordinates the parse-then-import workflow in the background via ActiveJob.

The problem

The user clicks "Import". The job goes into the queue. And the page sits there.

No progress indicator. No feedback. No way to know if the import is 10% done, 90% done, or failed entirely. The only option is to refresh and check the status column. For a 50,000-row CSV that takes two minutes, that is a terrible experience.

Before:  click "Import" → spinner → ??? → refresh → refresh → done (maybe)
After:   click "Import" → live progress bar 0%...50%...100% → auto-reload with results

Rails ships ActionCable, so we use it. The challenge is keeping the broadcasting layer decoupled from the Orchestrator and providing a Stimulus integration that works without the host developer writing any JavaScript.

What we're building

Here is the flow from server to browser:

Orchestrator#import!
  |
  |-- for each record:
  |     Broadcaster#progress(current, total)
  |       --> ActionCable.server.broadcast("data_porter/imports/42", { status: :processing, percentage: 65, ... })
  |             --> ImportChannel streams to subscriber
  |                   --> Stimulus progress_controller updates the bar
  |
  |-- on success:
  |     Broadcaster#success
  |       --> { status: :success }
  |             --> Stimulus reloads the page
  |
  |-- on failure:
        Broadcaster#failure(message)
          --> { status: :failure, error: "..." }
                --> Stimulus reloads the page

Three objects, three layers. The Broadcaster knows how to format messages. The ImportChannel knows how to route them. The Stimulus controller knows how to render them. None of them knows about the others' internals.

Implementation

Step 1 -- The Broadcaster service

The Broadcaster is a plain Ruby object that wraps ActionCable.server.broadcast with import-specific semantics:

# lib/data_porter/broadcaster.rb
module DataPorter
  class Broadcaster
    def initialize(import_id)
      prefix = DataPorter.configuration.cable_channel_prefix
      @channel = "#{prefix}/imports/#{import_id}"
    end

    def progress(current, total)
      percentage = ((current.to_f / total) * 100).round
      broadcast(status: :processing, percentage: percentage, current: current, total: total)
    end

    def success
      broadcast(status: :success)
    end

    def failure(message)
      broadcast(status: :failure, error: message)
    end

    private

    def broadcast(message)
      ActionCable.server.broadcast(@channel, message)
    end
  end
end

Three public methods, three states the browser cares about: work is in progress, work succeeded, or work failed. The percentage is computed server-side so the client just sets a CSS width. The channel name uses a configurable prefix (data_porter/imports/42) to avoid collisions with the host app's channels.

The Broadcaster plugs into the Orchestrator's import loop:

# Inside Orchestrator#import_records (conceptual)
broadcaster = Broadcaster.new(@data_import.id)
importable.each_with_index do |record, index|
  persist_record(record, context, results)
  broadcaster.progress(index + 1, importable.size)
end
broadcaster.success

One progress call per record. ActionCable broadcasts are cheap (in-memory pub/sub with the async adapter, Redis PUBLISH with Redis), and the Stimulus controller handles them idempotently -- it just sets a CSS width, so skipped frames cause no harm.

Step 2 -- The ImportChannel

The channel is the thinnest class in the entire engine:

# app/channels/data_porter/import_channel.rb
module DataPorter
  class ImportChannel < ActionCable::Channel::Base
    def subscribed
      prefix = DataPorter.configuration.cable_channel_prefix
      stream_from "#{prefix}/imports/#{params[:id]}"
    end
  end
end

That is the entire file. The channel constructs the same stream name the Broadcaster uses -- same prefix, same format. No authorization logic here: by the time the user sees the progress bar, the controller has already authorized access to that import.

Step 3 -- The Stimulus progress controller

On the browser side, a Stimulus controller subscribes to the channel and updates the DOM:

// app/javascript/data_porter/progress_controller.js
import { Controller } from "@hotwired/stimulus"
import { createConsumer } from "@rails/actioncable"

export default class extends Controller {
  static targets = ["bar", "text"]
  static values = { id: Number }

  connect() {
    this.subscription = createConsumer().subscriptions.create(
      { channel: "DataPorter::ImportChannel", id: this.idValue },
      {
        received: (data) => {
          if (data.status === "processing") {
            this.updateProgress(data.percentage)
          } else {
            window.location.reload()
          }
        }
      }
    )
  }

  updateProgress(percentage) {
    if (this.hasBarTarget) {
      this.barTarget.style.width = `${percentage}%`
      this.textTarget.textContent = `${percentage}%`
    }
  }

  disconnect() {
    this.subscription?.unsubscribe()
  }
}

The controller declares two targets (bar and text) and one value (id). The corresponding HTML looks like this:

<div data-controller="data-porter--progress" data-data-porter--progress-id-value="42">
  <div data-data-porter--progress-target="bar" style="width: 0%"></div>
  <span data-data-porter--progress-target="text">0%</span>
</div>

The logic is two branches: if processing, update the bar; for anything else (success or failure), reload the page. The reload is the simplest possible terminal action -- the server-rendered page shows the final state, no client-side state management needed.

A few design choices worth noting:

createConsumer() instead of a shared consumer -- in an engine context, we cannot assume the host app exports one. ActionCable deduplicates connections internally.
hasBarTarget guard -- degrades gracefully during Turbo transitions where the DOM might be partially rendered.
window.location.reload() on completion -- the engine does not need to ship Turbo Stream templates for the result screen. The server renders it once.

Step 4 -- Configuration

The channel prefix defaults to "data_porter" and is overridable in the initializer:

# config/initializers/data_porter.rb
DataPorter.configure do |config|
  config.cable_channel_prefix = "my_app_imports"
end

This prevents channel name collisions if the host app runs multiple engines that use ActionCable.

Decisions & tradeoffs

Decision	We chose	Over	Because
Real-time transport	ActionCable (WebSockets)	Polling or SSE	Rails ships ActionCable; no extra dependencies, integrates with existing auth, bidirectional even though we only need server-to-client
Broadcast granularity	One message per record	Batched (every N records) or throttled (every N seconds)	Simplicity; ActionCable broadcasts are cheap, and the Stimulus controller handles high-frequency updates idempotently by just setting a CSS width
Completion behavior	`window.location.reload()`	Turbo Stream partial updates	The engine cannot predict what the host app's result page looks like; a full reload lets the server render the final state with its own layout and components
Channel authorization	None (deferred to controller)	`reject` in `subscribed` based on user ownership	The engine does not know the host's auth system; by the time the user sees the progress bar, the controller has already authorized access

Recap

The Broadcaster is a plain Ruby service that wraps ActionCable.server.broadcast with three semantic methods: progress, success, and failure. It constructs the channel name from a configurable prefix and the import ID.
The ImportChannel is a one-method ActionCable channel that streams from the same channel name the Broadcaster writes to. It contains no authorization logic -- that responsibility stays in the controller layer.
The Stimulus progress controller subscribes on connect, updates a progress bar on processing messages, reloads the page on success or failure, and cleans up the subscription on disconnect.
The cable_channel_prefix configuration option prevents channel name collisions with the host app and gives operations teams control over naming.

Next up

The import now runs in the background and pushes live progress to the browser. But the progress bar needs a page to live on, and right now the engine has no UI. In part 9, we build the view layer with Phlex and Tailwind -- auto-generated preview tables from the target DSL, status badges, and scoped CSS that does not leak into the host app.

This is part 8 of the series "Building DataPorter - A Data Import Engine for Rails". Previous: The Orchestrator | Next: Building the UI with Phlex & Tailwind

GitHub: SerylLns/data_porter | RubyGems: data_porter

DEV Community: Seryl Lns

Building a Rails Engine #16 --Publishing to RubyGems & Retrospective

Publishing to RubyGems & Retrospective

Context

Publishing the gem

The gemspec

Versioning

The release workflow

Documentation

What we built

Lessons learned

TDD without a dummy app

StoreModel gotchas

Phlex in an engine: plain vs text

Testing patterns: controllers, channels, JS

What is next

Final reflection

Building a Rails Engine #15 -- ERB Views Meet Phlex Components

ERB View Templates: Composing Phlex Components

Context

The missing attachment

Upgrading the test infrastructure

Rendering Phlex in ERB

The index template

The new import form

The show template: status-driven composition

The CSS stylesheet

Testing views in a Rails 8 engine

Decisions & tradeoffs

Recap

Next up

Building a Rails Engine #14 -- Dry Run: Validate Before You Import

Dry Run: Validate Before You Import

Context

Why two validation layers

The dry_run_enabled DSL flag

The dry_running status

The dry_run! flow

The StoreModel dirty tracking gotcha

Job, controller, and route

Testing

Decisions & tradeoffs

Recap

Next up

Building a Rails Engine #13 -- How to test a mountable Rails engine without a dummy app

Testing a Rails Engine with RSpec

Context

The problem

What we are building

Implementation

Step 1 -- The in-memory database

Step 2 -- Load paths and the ApplicationController stub

Step 3 -- Testing models and StoreModel objects

Step 4 -- Testing controllers structurally

Step 5 -- Testing Phlex components

Step 6 -- Testing generators

Step 7 -- Testing JavaScript from Ruby specs

Step 8 -- The anonymous target class pattern

Decisions & tradeoffs

The TDD workflow

Recap

Next up

Building a Rails Engine #12 -- Beyond CSV: JSON and API Sources

Beyond CSV: JSON and API Sources

Context

DataPorter Sources Architecture — Explaina

Why multiple sources?

The JSON source

The API source

The ApiConfig DSL pattern

Dispatch via Sources.resolve

The TDD approach

Decisions & tradeoffs

Recap

Next up

Turn Messy WhatsApp Messages Into Structured JSON-Reliably

Designing Deterministic Outputs From Unstructured Messages

The Problem

Step 1: Define Your Schema

Step 2: Extract, Don't Converse

Phlex in an engine: `plain` vs `text`

The `dry_run_enabled` DSL flag

The `dry_running` status

The `dry_run!` flow

The `dp-` CSS prefix strategy