DEV Community: Tony Rowan

Mock User Agent in View Component Specs

Tony Rowan — Fri, 17 Jan 2025 11:26:32 +0000

Recently, I implemented a couple of ViewComponents to display links to our app in the app stores. These ViewComponents contained in a single place: the appropriate logo for app store, the link to the app in said app store, descriptions to make the links accessible and click tracking.

The links also needed to support a small piece of behaviour. We needed the links to only be displayed on what we deemed an appropriate platform. Explicitly, the link to the Apple App Store shouldn't be rendered on Android devices and the link to the Google Play Store shouldn't be rendered on iOS devices.

To keep things very simple and self-contained, we can use the browser gem and access the view helper it provides directly from the view component templates to detect in the component should render anything.

The resulting view components look a little something like this. Descriptions and click tracking have been omitted for brevity, we're not looking at those in this post.

# app/components/apple_app_store_link_component.rb
class AppleAppStoreLinkComponent < ViewComponent::Base
end

<!-- app/components/apple_app_store_link_component.html.erb -->
<% unless helpers.browser.platform.android? %>
  <%= link_to "<link_to_the_app_on_the_apple_app_store>" do %>
    <%= image_tag("apple_app_store.svg") %>
  <% end %>
<% end %>

Testing these components should be simple. We should be able to set the User Agent String to match an android device and an ios device and assert that the component renders (or not) as appropriate.

We're using RSpec and we already have :android and :ios tags that are used in system test hooks to set the relevant User Agent String on the Rack driver.

config.before(:each, type: :system, android: true) do
  page.driver.browser.current_session.header("User-Agent", android_user_agent_string)
end

config.before(:each, type: :system, ios: true) do
  page.driver.browser.current_session.header("User-Agent", ios_user_agent_string)
end

Obviously these hooks would not work as-is, we need to also target component tests.

config.before(:each, type: :component, android: true) do
  page.driver.browser.current_session.header("User-Agent", android_user_agent_string)
end

config.before(:each, type: :component, ios: true) do
  page.driver.browser.current_session.header("User-Agent", ios_user_agent_string)
end

Now we can write our simple and descriptive component specs.

RSpec.describe AppleAppStoreLinkComponent, type: :component do
  it "does not render the link on Android devices", :android do
    render_inline(described_class.new)

    expect(page).not_to have_css(apple_app_store_selector)
  end

  it "renders the link on iOS devices", :ios do
    render_inline(described_class.new)

   expect(page).to have_css(apple_app_store_selector)
  end

  private

  def apple_app_store_selector
    "a[href='<link_to_the_app_on_the_apple_app_store>'] img[src*='apple_app_store']"
  end
end

Unfortunately, this doesn't work. The Capybara driver used in component specs is very simple and doesn't support the full API supported by, say, the Rack driver. Here, we get an error.

  NoMethodError:
    undefined method `driver' for an instance of Capybara::Node::Simple

We need another way to set the User Agent String for component tests.

Looking through the ViewComponent Testing guide didn't help. There were sections on setting the current request parameters, but nothing about how to set
arbitrary headers. GPT and Copilot were also of no help, since they just hallucinated non-existent APIs.

Before giving up and using a full-on system test for this behaviour, which felt very heavy for a simple case of user agent detection, I looked through the extensive API documentation. I found vc_test_request, a way to access and modify the request object directly in tests - the docs even used the User Agent header as an example.

This was perfect! I plugged it into the hook and everything worked as expected.

config.before(:each, type: :component, android: true) do
  vc_test_request.env["HTTP_USER_AGENT"] = android_user_agent_string
end

config.before(:each, type: :component, ios: true) do
  vc_test_request.env["HTTP_USER_AGENT"] = ios_user_agent_string
end

I wrote this post because it wasn't immediately clear to me from the documentation that this testing behaviour was available and I thought maybe others would find it useful.

Thanks for reading!

Photo by berenice melis on Unsplash

How To: Implement an Android Smart App Banner

Tony Rowan — Fri, 23 Jun 2023 14:30:26 +0000

So there isn't an Android Smart App Banner like the one for iOS but there is the Native App Install Prompt, which is about as much as we get natively on Android. Sorry for the bait-and-switch, there...

It looks like this.

Tapping the banner brings up a dialog like this.

Tapping install in that dialog takes you to the Play Store.

So, How Do You Implement It?

Unfortunately, the documentation for the prompt isn't very good. It skips over important information and talks about a very similar prompt for PWAs without ever clarifying which bits matter to what prompts.

To get this banner working for Native Android Apps all you have to do is add a manifest file with the following properties:

short_name: The name of your app/brand/company/whatever. It's shown on the initial banner.
name: A longer form name for your app/brand/company/whatever. It's shown on the dialog.
icons: An array of objects describing the icons to show on the banner and dialog.
"prefer_related_applications": true. If this key is missing or not set to true, the banner won't show up.
related_applications: A list of objects that describe the app to get the user to install. To get the banner to show up on Android, the list must contain an object with a key called platform set to play and a key called id, which is set to the ID of the app in the Play Store.

You then have to ensure that the manifest is declared in your HTML.

<link rel="manifest" href="/manifest.json" />

That's it, the prompt will now show up. You don't need to listen for events or show buttons or anything. I repeat:

All you need to do to get the banner to show up is to add a manifest file with the required properties and declare it in your HTML.

Example

{
  "short_name": "Cool App",
  "name": "Cool App - You're going to love it",
  "icons": [
    {
      "src": "/android-chrome-192x192.png",
      "type": "image/png",
      "sizes": "192x192"
    },
    {
      "src": "/android-chrome-512x512.png",
      "type": "image/png",
      "sizes": "512x512"
    }
  ],
  "prefer_related_applications": true,
  "related_applications": [
    {
      "platform": "play",
      "id": "com.example.cool_app"
    }
  ]
}

I Don't See the Prompt

Unfortunately, even if you do this, the prompt still will not likely show up. As noted in the documentation, there are several other factors that Chrome uses before it will trigger the prompt. This includes:

The user needs to not have the app already installed (obviously...otherwise you'd just add a static banner in your HTML, right?)
The page must be served over HTTPS (I hope your production traffic is already doing this)
The user needs to be on a mobile device. I haven't seen this listed anywhere, but I've only ever gotten it to trigger using mobile Chrome. It feels kind of obvious when you think about it.
The user needs to meet some kind of interaction requirement. This is pretty vague and basically means the prompt won't show up the first time (probably not the second or third time either) the user visits your site. They will only see it if they are interacting with the domain quite often.

These requirements can make testing whether you've set up your manifest correctly quite difficult. Fortunately, there are a couple of Chrome flags you can use to make things easier.

Bypass User Engagement Checks

If you enable #bypass-app-banner-engagement-checks, then the banner will show up on the first (and every) visit to your site, as long as the other criteria are met.

Insecure Origins Treated As Secure

Since you need to be on a mobile device, it might be hard to visit your site running in local development mode without paying for a service like ngrok. On Mac OS, at least, you can visit your site running locally from a mobile device on the same WiFi network by simply visiting the IP for your Mac (found in the wifi settings panel). You can then tell Chrome to treat that IP address as secure by adding it to the #unsafely-treat-insecure-origin-as-secure flag.

Advanced Usage

You don't have to serve your manifest file as a static file. If you want to make use of asset helpers or translations, you can back your manifest file with an endpoint. This example is ruby based, but the same approach works wherever.

<link rel="manifest" href="/webapp_manifest" />

# routes.rb
resource :webapp_manifest, only: :show

# app/controllers/webapp_manifests_controller.rb
class WebappManifestsController
  def show
    render json: {
      name: I18n.t("long_app_name"),
      short_name: I18n.t("short_app_name"),
      prefer_related_applications: true,
      related_applications: {
        id: AppIds::PLAY_STORE_APP_ID,
        platform: "play"
      },
      icons: [
        {
          src: asset_url("app_store_icon_512")
          type: "image/png",
          sizes: "512x512"
        },        
        {
          src: asset_url("app_store_icon_192")
          type: "image/png",
          sizes: "192x192"
        },
      ]
    }
  end
end

Conclusion

I hope you find this post useful and less confusing than the official documentation.

I've now implemented this for three different sites for three different companies over the last 5 years and each time I wished for a post like this.

So, this time, I wrote it down. This has mainly been for me more than anything 😉

Cover Photo by Towfiqu barbhuiya on Unsplash

A Deep Dive into Memory Leaks in Ruby

Tony Rowan — Wed, 17 Aug 2022 11:28:09 +0000

In the first part of this two-part series on memory leaks, we looked at how Ruby manages memory and how Garbage Collection (GC) works.

You might be able to afford powerful machines with more memory, and your app might restart often enough that your users don't notice, but memory usage matters.

Allocation and Garbage Collection aren't free. If you have a leak, you spend more and more time on Garbage Collection instead of doing what you built your app to do.

In this post, we'll look deeper into the tools you can use to discover and diagnose a memory leak.

Let's continue!

Finding Leaks in Ruby

Detecting a leak is simple enough. You can use GC, ObjectSpace, and the RSS graphs in your APM tool to watch your memory usage increase. But just knowing you have a leak is not enough to fix it. You need to know where it is coming from. Raw numbers can't tell you that.

Fortunately, the Ruby ecosystem has some great tools to attach context to those numbers. Two are memory-profiler and derailed_benchmarks.

`memory_profiler` in Ruby

The memory_profiler gem offers a very simple API and a detailed (albeit a little overwhelming) allocated and retained memory report — that includes the classes of objects that are allocated, their size, and where they were allocated. It's straightforward to add to our leaky program.

# leaky.rb
require "memory_profiler"

an_array = []

report = MemoryProfiler.report do
  11.times do

    1000.times { an_array << "A" + "B" + "C" }
    puts "Array is #{an_array.size} items long"
  end

  GC.start
end

report.pretty_print

Outputting a report that looks similar to this.

Total allocated: 440072 bytes (11001 objects)
Total retained:  440072 bytes (11001 objects)

allocated memory by gem
-----------------------------------
    440072  other

allocated memory by file
-----------------------------------
    440072  ./leaky.rb

allocated memory by location
-----------------------------------
    440000  ./leaky.rb:9
        72  ./leaky.rb:10

allocated memory by class
-----------------------------------
    440000  String
        72  Thread::Mutex

allocated objects by gem
-----------------------------------
     11001  other

allocated objects by file
-----------------------------------
     11001  ./leaky.rb

allocated objects by location
-----------------------------------
     11000  ./leaky.rb:9
         1  ./leaky.rb:10

allocated objects by class
-----------------------------------
     11000  String
         1  Thread::Mutex

retained memory by gem
-----------------------------------
    440072  other

retained memory by file
-----------------------------------
    440072  ./leaky.rb

retained memory by location
-----------------------------------
    440000  ./leaky.rb:9
        72  ./leaky.rb:10

retained memory by class
-----------------------------------
    440000  String
        72  Thread::Mutex

retained objects by gem
-----------------------------------
     11001  other

retained objects by file
-----------------------------------
     11001  ./leaky.rb

retained objects by location
-----------------------------------
     11000  ./leaky.rb:9
         1  ./leaky.rb:10

retained objects by class
-----------------------------------
     11000  String
         1  Thread::Mutex


Allocated String Report
-----------------------------------
     11000  "ABC"
     11000  ./leaky.rb:9


Retained String Report
-----------------------------------
     11000  "ABC"
     11000  ./leaky.rb:9

There is a lot of information here, but generally, the
allocated objects by location and retained objects by location sections can be the most useful when looking for leaks. These are the file locations that allocate objects, ordered by the number of allocated objects.

allocated objects are all objects allocated (created) within the report block.
retained objects are objects that have not been garbage collected by the end of the report block. We forced a GC run before the end of the block so we could see the leaked objects more clearly.

Be careful about trusting the retained object counts. They depend heavily on what portion of the leaking code is within the report block.

For example, if we move the declaration of an_array into the report block, we might be fooled into thinking the code isn't leaky.

# leaky.rb
require "memory_profiler"

report = MemoryProfiler.report do
  an_array = []

  11.times do

    1000.times { an_array << "A" + "B" + "C" }
    puts "Array is #{an_array.size} items long"
  end

  GC.start
end

report.pretty_print

The top of the resulting report won't report many retained objects (just the report itself).

Total allocated: 529784 bytes (11002 objects)
Total retained:  72 bytes (1 objects)

`derailed_benchmarks` in Ruby

The derailed_benchmarks gem is a suite of very useful tools for all kinds of performance work, primarily aimed at Rails apps. For finding leaks, we want to look at perf:mem_over_time, perf:objects, and perf:heap_diff.

These tasks work by sending curl requests to a running app, so we can't add them to our little leaky program. Instead, we'll need to set up a small Rails app with an endpoint that leaks memory, then install the derailed_benchmarks on that app.

# Create a rails app with no database
rails new leaky --skip-active-record --minimal

## Add derailed benchmarks
cd leaky
bundle add derailed_benchmarks

# config/routes.rb
Rails.application.routes.draw do
  root "leaks#index"
end

# app/controllers/leaks_controller.rb
class LeaksController < ApplicationController
  def index
    1000.times { $an_array << "A" + "B" + "C" }

    render plain: "Array is #{$an_array.size} items long"
  end
end

# config/initializers/array.rb
$an_array = []

You should now be able to boot the app with bin/rails s. You'll be able to curl an endpoint that leaks on each request.

$ curl http://localhost:3000

Array is 1000 items long

$ curl http://localhost:3000

Array is 2000 items long

We can now use derailed_benchmarks to see our leak in action.

`perf:mem_over_time`

This will show us memory use over time (similarly to how we watched the memory growth of our leaky script with watch and ps).

Derailed will boot the app in production mode, repeatedly hit an endpoint (/ by default), and report the memory usage. If it never stops growing, we have a leak!

$ TEST_COUNT=10000 DERAILED_SKIP_ACTIVE_RECORD=true \
  bundle exec derailed exec perf:mem_over_time

Booting: production
Endpoint: "/"
PID: 4417
104.33984375
300.609375
455.578125
642.69140625
751.6953125

Note: Derailed will boot the Rails app in production mode to perform the tests. By default, it will also require rails/all first. Since we don't have a database in this app, we need to override this behavior with DERAILED_SKIP_ACTIVE_RECORD=true.

We can run this benchmark against different endpoints to see which one/s (if any) leak.

`perf:objects`

The perf:objects task uses memory_profiler under the hood so the produced report will look familiar.

$ TEST_COUNT=10 DERAILED_SKIP_ACTIVE_RECORD=true \
  bundle exec derailed exec perf:objects

Booting: production
Endpoint: "/"
Running 10 times
Total allocated: 2413560 bytes (55476 objects)
Total retained:  400000 bytes (10000 objects)

# The rest of the report...

This report can help narrow down where your leaked memory is being allocated. In our example, the report's last section — the Retained String Report — tells us exactly what our problem is.

Retained String Report
-----------------------------------
     10000  "ABC"
     10000  /Users/tonyrowan/playground/leaky/app/controllers/leaks_controller.rb:3

We've leaked 10,000 strings containing "ABC" from the LeaksController on line 3. In a non-trivial app, this report would be significantly larger and contain retained strings that you want to retain — query caches, etc. — but this and the other 'by location' sections should help you narrow down your leak.

`perf:heap_diff`

The perf:heap_diff benchmark can help if the report from perf:objects is too complex to see where your leak is coming from.

As the name suggests, perf:heap_diff produces three heap dumps and calculates the difference between them. It creates a report that includes the types of objects retained between dumps and the location that allocated them.

$ DERAILED_SKIP_ACTIVE_RECORD=true bundle exec derailed exec perf:heap_diff

Booting: production
Endpoint: "/"
Running 1000 times
Heap file generated: "tmp/2022-06-15T11:08:28+01:00-heap-0.ndjson"
Running 1000 times
Heap file generated: "tmp/2022-06-15T11:08:28+01:00-heap-1.ndjson"
Running 1000 times
Heap file generated: "tmp/2022-06-15T11:08:28+01:00-heap-2.ndjson"

Diff
====
Retained STRING 999991 objects of size 39999640/40008500 (in bytes) at: /Users/tonyrowan/playground/leaky/app/controllers/leaks_controller.rb:3
Retained STRING 2 objects of size 148/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:265
Retained STRING 1 objects of size 88/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:266
Retained DATA 1 objects of size 72/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/3.1.0/objspace.rb:87
Retained IMEMO 1 objects of size 40/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/3.1.0/objspace.rb:88
Retained IMEMO 1 objects of size 40/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:259
Retained IMEMO 1 objects of size 40/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:260
Retained FILE 1 objects of size 8432/40008500 (in bytes) at: /Users/tonyrowan/.asdf/installs/ruby/3.1.2/lib/ruby/gems/3.1.0/gems/derailed_benchmarks-2.1.1/lib/derailed_benchmarks/tasks.rb:266

Run `$ heapy --help` for more options

You can also read Tracking a Ruby memory leak in 2021 to understand better what's going on.

The report points us exactly where we need to go for our leaky baby app. At the top of the diff, we see 999991 retained string objects allocated from the LeaksController on line 3.

Leaks in Real Ruby and Rails Apps

Hopefully, the examples we've used so far have never been put into real-life apps — I hope no one intends to leak memory!

In non-trivial apps, memory leaks can be much harder to track down. Retained objects are not always bad — a cache with garbage collected items would not be of much use.

There is something common between all leaks, though. Somewhere, a root-level object (a class/global, etc.) holds a reference to an object.

One common example is a cache without a limit or an eviction policy. By definition, this will leak memory since every object put into the cache will remain forever. Over time, this cache will occupy more and more of the memory of an app, with a smaller and smaller percentage of it actually in use.

Consider the following code that fetches a high score for a game. It's similar to something I've seen in the past. This is an expensive request, and we can easily bust the cache when it changes, so we want to cache it.

class Score < ApplicationModel
  def self.user_high_score(game, user)
    @scores = {} unless @scores

    if (score = @scores["#{game.id}:#{user.id}"])
      score
    else
      Score.where(game: game, user: user).order(:score).first.tap do |score|
        @scores["#{game.id}:#{user.id}"] = score
      end
    end
  end

  def self.save_score(game, user, raw_score)
    score = create!(game: game, user: user, score: raw_score)

    if raw_score > user_high_score(game, user).score
      @scores["#{game.id}:#{user.id}"] = score
    end
  end
end

The @scores hash is completely unchecked. It will grow to hold every single high score for every user — not ideal if you have a lot of either.

In a Rails app, we would probably want to use Rails.cache with a sensible expiry (a memory leak in Redis is still a memory leak!) instead.

In a non-Rails app, we want to limit the hash size, evicting the oldest or least recently used items. LruRedux is a nice implementation.

A more subtle version of this leak is a cache with a limit, but whose keys are of arbitrary size. If the keys themselves grow, so too will the cache. Usually, you won't hit this. But, if you're serializing objects as JSON and using that as a key, double-check that you're not serializing things that grow with usage as well — such as a list of a user's read messages.

Circular References

Circular references can be garbage collected. Garbage Collection in Ruby uses the "Mark and Sweep" algorithm. During their presentation introducing variable width allocation, Peter Zhu and Matt Valentine-House gave an excellent explanation of how this algorithm works.

Essentially, there are two phases: marking and sweeping.

In the marking phase, the garbage collector starts at root objects (classes, globals, etc.), marks them, and then looks at their referenced objects.

It then marks all of the referenced objects. Referenced objects that are already marked are not looked at again. This continues until there are no more objects to look at — i.e., all referenced objects have been marked.

The garbage collector then moves on to the sweeping phase. Any object not marked is cleaned up.

Therefore, objects with live references can still be cleaned up. As long as a root object does not eventually reference an object, it will be collected. In this way, clusters of objects with circular references can still be garbage collected.

Application Performance Monitoring: The Event Timeline and Allocated Objects Graph

As mentioned in the first part of this series, any production-level app should use some form of Application Performance Monitoring (APM).

Many options are available, including rolling your own (only recommended for larger teams). One key feature you should get from an APM is the ability to see the number of allocations an action (or background job) makes. Good APM tools will break this down, giving insight into where allocations come from — the controller, the view, etc.

This is often called something like an 'event timeline.' Bonus points if your APM allows you to write custom code that further breaks down the timeline.

Consider the following code for a Rails controller.

class LeaksController < ApplicationController
  before_action :leak

  def index
    @leaks = $leak.sample(100)
  end


  private

  def leak
    1000.times { $leak << Leak.new }
  end
end

When reported by an APM, the 'event timeline' might look something like the following screenshot from AppSignal.

This can be instrumented so we can see which part of the code makes the allocations in the timeline. In real apps, it is probably going to be less obvious from the code 😅

class LeaksController < ApplicationController
  before_action :leak

  def index
    Appsignal.instrument('leak.fetch_leaks') do
      @leaks = $leak.sample(100)
    end
  end


  private

  def leak
    return unless params[:leak]

    Appsignal.instrument('leak.create_leaks') do
      1000.times { $leak << Leak.new }
    end
  end
end

Here's an example of an instrumented event timeline, again from AppSignal:

Knowing where to instrument can often be difficult to grasp. There's no substitute for really understanding your application's code, but there are some signals that can serve as 'smells'.

If your APM surfaces GC runs or allocations over time, you can look for spikes to see if they match up with certain endpoints being hit or certain running background jobs. Here's another example from AppSignal's Ruby VM magic dashboard:

By looking at allocations in this way, we can narrow down our search when
looking into memory problems. This makes it much easier to use tools like
memory_profiler and derailed_benchmarks efficiently.

Read about the latest additions to AppSignal's Ruby gem, like allocation and GC stats tracking.

Wrapping Up

In this post, we dived into some tools that can help find and fix memory leaks, including memory_profiler, derailed_benchmarks, perf:mem_over_time, perf:objects, perf:heap_diff, the event timeline and allocated objects graph in AppSignal.

I hope you've found this post, alongside part one, useful in diagnosing and sorting out memory leaks in your Ruby app.

How to Track Down Memory Leaks in Ruby

Tony Rowan — Wed, 10 Aug 2022 12:51:42 +0000

A memory leak is an unintentional, uncontrolled, and unending increase in memory usage. No matter how small, eventually, a leak will cause your process to run out of memory and crash. Even if you periodically restart your app to avoid this crash (no judgment, I've done that!), you still suffer the performance implications of a memory leak.

In this post, the first of a two-part series on memory leaks, we'll start by looking at how Ruby manages memory, how Garbage Collection (GC) works, and how to find a leak.

In the second part, we'll take a deeper dive into tracking down leaks.

Let's get started!

Ruby Memory Management

Ruby objects are stored on the heap, and each object fills one slot on the heap.

Prior to Ruby 3.1, all slots on the heap were the same size — 40 bytes, to be exact. Objects too large to fit in a slot were stored outside the heap. Each slot included a reference to where objects were moved.

In Ruby 3.1, variable width allocation for String objects was merged. Soon, variable width allocation will be the norm for all object types.

Variable width allocation aims to improve performance by improving cache locality — all the information of an object will be stored in one place rather than across two memory locations.

It should also simplify (some parts) of memory management. At the moment, there are two 'heaps':

The Ruby heap (or GC heap) that stores smaller Ruby objects.
The C heap (or malloc/transient heap) that stores larger objects.

Once variable width allocation is the norm, there should be no need for the latter heap.

The heap starts at a given size (10,000 slots by default) and objects are assigned to free slots as they are created. When Ruby tries to create an object and there are no free slots available, Garbage Collection (GC) occurs to make some free slots available.

If there are too few free slots after GC, the heap will be expanded (more on this a little later).

Here are the factors you can control, alongside their environment variables:

Initial size of the heap - RUBY_GC_HEAP_INIT_SLOTS
Number of free slots that should be available after GC occurs - RUBY_GC_HEAP_FREE_SLOTS
Amount the heap is expanded by -RUBY_GC_HEAP_GROWTH_FACTOR

Garbage Collection in Ruby

Garbage Collection in Ruby 'stops the world' — no other process occurs when GC occurs. Garbage Collection in Ruby
(since 2.1) is also generational, meaning that the garbage collector has two modes:

Minor GC - inspects 'young' objects (objects created recently)
Major GC - inspects 'old' objects as well as 'young' objects (all the objects)

Note: An 'old' object has survived 3 GC runs, major or minor.

When the heap is full, minor GC is invoked first. If it can't free up enough slots to be below the limit, major GC will be invoked. Only then, if there are still not enough free slots, will the heap be expanded.

Major GC is more expensive than minor GC because it looks at more objects.

The theory behind why generational GC is more performant is that objects usually fall into two categories:

Objects that are allocated and then quickly go out of scope. In a Rails app, models fetched from the DB to render a page will go out of scope when the request ends.
Objects that are allocated and kept around for a long time. Classes and caches are likely to still be in use throughout the lifetime of an app.

Major GC will also run after minor GC if the number of old objects is above a certain threshold, even if there are sufficient free slots. This limit increases as the size of the heap grows and can be controlled by the RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR environment variable.

When you have a leak, you create objects that can't be cleaned up — more and more old objects. This means that major (expensive) GC will run much more often than it should. Since nothing else runs when GC is running, this is time that you waste.

I've left some links at the end of this article for further reading on memory layout and the garbage collector in Ruby.

What Does A Memory Leak Look Like in Ruby?

You can see a memory leak using simple tools available on any
Unix system. Take the following code as an example.

# leaky.rb

an_array = []

loop do
  1000.times { an_array << "A" + "B" + "C" }
  puts an_array.size
  sleep 1
end

To say this code 'leaks' is a little unfair — all it does is leak! — but it serves our purposes.

We can observe the leak quite simply from the command line by running this program in one terminal and watch-ing the memory increase over time with ps.

# In terminal one
$ ruby ./leaky.rb

# In terminal two
$ watch ps -p `pgrep -f "ruby ./leaky.rb"` -o pmem,pcpu,rss,args

The pgrep -f "ruby ./leaky.rb" finds the process ID for us, so that we can restrict the ps output to only the process we're interested in. As you may be able to guess, it's like grep for processes.

The watch tool allows us to poll the output of a given command and update it in place, giving us a live dashboard within our terminal.

You'll get output like this, which updates every couple of seconds.

Every 2.0s: ps -p 50866 -o pmem,pcpu,rss,args

%MEM  %CPU    RSS ARGS
 0.2   4.1 163408 /Users/tonyrowan/.asdf/installs/ruby/3.1.1/bin/ruby ./leaky.rb

You should see the %MEM and RSS increasing. They are:

%MEM - The amount of memory the process uses as a percentage of memory on the host machine.
RSS (resident set size) - The amount of RAM the process uses in bytes.

This basic OS-only information is enough to spot if you have a leak — if the memory keeps going up, it means you do!

Find Ruby Leaks with the Garbage Collector Module

We can also detect leaks within Ruby code itself with the GC module.

# leaky.rb

GC.disable # Only run GC when manually called

an_array = []

loop do
  1000.times { an_array << "A" + "B" + "C" }
  puts "Array is #{an_array.size} items long"

  GC.start # Run a major GC - use full_mark: false for minor GC
  puts "There are #{GC.stat(:heap_live_slots)} live objects"

  sleep 1
end

The GC.stat method will return a hash with a lot of useful information. Here, we're interested in :heap_live_slots, which is the number of slots on the heap that are in use. That's the opposite of :heap_free_slots.

At the end of the loop, we force a major GC and print out the number of used slots, i.e., the number of objects that remain after GC.

When we run our little program, we see this increase ad infinitum. We have a leak! We could also have used GC.stat(:old_objects) to the same effect.

While the GC module can be used to see if we have a leak and (if you're smart with your puts statements) where the leak might be occurring, we can see the type of objects that might be leaking with the ObjectSpace module.

# leaky.rb

GC.disable # Only run GC when manually called

an_array = []

loop do
  1000.times { an_array << "A" + "B" + "C" }
  puts "Array is #{an_array.size} items long"

  GC.start # Run a major GC - use full_mark: false for minor GC
  pp ObjectSpace.count_objects

  sleep 1
end

The ObjectSpace.count_objects method returns a hash with the counts of live objects. T_STRING, for instance, is the number of strings live in memory. For our rather leaky program, this value increases with each loop, even after GC. We can see that we are leaking string objects.

Application Performance Monitoring in Production with AppSignal

While playing with ps and GC can be a sensible route for toy projects — they're also fun and informative to use! — I would not recommend them as your memory leak detection solution in production apps.

This is where you would use an Application Performance Monitoring (APM) tool. If you're a very large company, you can build these yourself. For smaller outfits, though, picking an APM off-the-shelf is the way to go. You do need to pay a monthly subscription, but the information they provide more than makes up for it.

For detecting memory leaks, you want to find server or process memory use (sometimes called RSS) graphs over time. Here's an example screenshot from AppSignal's 'process memory usage' dashboard of a healthy app shortly after being deployed:

And here's an unhealthy app after deployment:

AppSignal will even surface Ruby VM stats like GC and heap slots, which can
give you an even clearer signal for a memory leak. If the number of live slots
keeps growing, you have a leak!

Wrap Up and Further Reading

In this post, we took a quick tour of Ruby's memory management and garbage collector. We then diagnosed how to discover a memory leak using Unix tools and Ruby's GC module.

Next time, we'll see how to use memory_profiler and derailed_benchmarks to find and fix leaks.

In the meantime, you can read more about the tools we used:

Additional further reading:

Happy coding, and see you next time!

P.S. If you'd like to read Ruby Magic posts as soon as they get off the press, subscribe to our Ruby Magic newsletter and never miss a single post!

Adding a Star Rating with Hotwire

Tony Rowan — Thu, 14 Jul 2022 08:46:21 +0000

Cover photo by GRANDBEAU on Unsplash

This post walks through the process of adding a rating system to a Turbo powered single page application. The application in question was built up in a previous post. If you want to follow along with the post, you can grab the repo at the starting point or just read the pull request.

What We're Building

In this walkthrough, we're going to be adding a basic star bar rating system to our movies single page application. You might be surprised to see that we can get a basic working version of this with no additional javascript whatsoever!

Add Ratings

The first step is to add ratings to our movies. Run the following in your terminal to create a migration file that will add a new column to our movies table.

bin/rails generate migration add_rating_to_movies rating:integer

Open the file and edit it to add a not-null constraint and a default value of 0.

class AddRatingToMovies < ActiveRecord::Migration[7.0]
  def change
    add_column :movies, :rating, :integer, null: false, default: 0
  end
end

We can also add a validation to our Movie model to ensure the rating stays within a sane bound that we can display on our star bar.

class Movie < ApplicationRecord
  validates :rating, numericality: { in: 0..5 }
end

Now we can run db:migrate to add the column. If you open
up a console with bin/rails console now you'll be able to update the rating on a movie e.g. Movie.first.update(rating: 5). Our validation will prevent you from setting the rating to something we can't display like -1 or 6.

Now we can update the view for our movies to display the rating.

It's pretty basic though, we can improve on that.

Add the Star Bar

The next step in creating our star bar is the stars! We're going to use icons from the wonderful heroicons which is a suite of beautiful SVG icons from the makers of TailwindCSS.

There's an icon called star that we'll use, copy the SVG contents of the icon and paste it into a file called app/assets/images/star.svg.

The best way to style an SVG icon (which is one of the big advantages of the format, in my opinion) is to embed the SVG in the HTML document (render it inline with the rest of the HTML) and from there apply CSS styles as you might expect. There's a gem called inline_svg that we'll make use of to make that easier.

Add to your Gemfile:

gem "inline_svg"

Run in your terminal:

bundle install

Now we can use the inline_svg_tag in our views.

To render our star bar we need to swap out the text rating we just added and replace it with this.

<div class="flex gap-x-3 mb-4">
  <% Movie::MAX_RATING.times do |n| %>
    <%= inline_svg_tag("star.svg",
      class: "w-8 h-8 #{star_rating_class(@movie, n)}") %>
  <% end %>
</div>

We going to render stars using the inline_svg_tag a total of
Movie::MAX_RATING number of times. This is a constant we've introduced for the maximum rating i.e. the maximum number of stars. To avoid duplication, we should reuse it in the validation.

class Movie < ApplicationRecord
  MAX_RATING = 5

  validates :rating, numericality: { in: 0..MAX_RATING }
end

We've also extracted a new helper method star_rating_class so we can keep the template readable by extracting the logic for which CSS classes the star should have, based on what rating the movie has. If the movie has a rating of 4 then
stars with index 0 through 3 should be solid yellow and the last star, with index 4, should be a grey outline.

# app/helpers/rating_helper.rb

module RatingHelper
  def star_rating_class(movie, index)
    if index < movie.rating
      "fill-yellow-400 stroke-yellow-400"
    else
      "fill-transparent stroke-gray-400"
    end
  end
end

Our star bar looks a lot nicer, but it's still not really doing anything. Let's change that next.

Submit a Rating

We want users to be able to click a star and apply that as a rating. We will need to add the update action to the routes for movies and implement that action in the MoviesController.

# config/routes.rb

Rails.application.routes.draw do
  resources :movies, only: %i[index show update]
end

# app/contollers/movies_controller.rb

def update
  @movie = Movie.find(params[:id])

  @movie.update(rating: params[:movie][:rating])

  render :show
end

The update action is fairly simple CRUD, but we only support updating the rating so there's no need for mass assignment. We're also rendering the show template at the end where we would normally redirect back to the show action. Redirecting would work here, but we're going to make use of not redirecting later when we add feedback.

In the template, we just need to turn the stars into buttons that will submit a form to the update endpoint PATCH movies/:id. We'll use the button_to template helper to do this. We'll set the method to :patch, the default is :post. We also need to add the index of the star
(plus 1 to account for the 0-based index) as a parameter in the form. In the HTML, this will be a hidden field with the name movie[rating].

<div class="flex gap-x-3 mb-4">
  <% Movie::MAX_RATING.times do |n| %>
    <%= button_to movie_path(@movie),
      method: :patch,
      params: { movie: { rating: n + 1 } } do %>
      <%= inline_svg_tag("star.svg",
        class: "
          w-8 h-8
          hover:fill-yellow-500 hover:stroke-yellow-500
          #{star_rating_class(@movie, n)}
        ") %>
    <% end %>
  <% end %>
</div>

If you try clicking stars now, you'll notice that it works perfectly, without any page refreshes. How? Turbo is taking over the form submission so that it is asynchronous, and when the response comes back and includes the :details turbo-frame, it swaps out the contents for the contents of the frame in the response.

In previous versions of Turbo, you could not return HTML from a form submission unless the status code was an error, successful submissions had to redirect. Fortunately for us, this is no longer a restriction.

User Feedback

The star bar is now working as intended, but it happens so seamlessly, it's not obvious that the user's rating has been persisted. We can improve this with a little user feedback. We can add a small 'toast' next to the star bar after the user rates a movie.

Let's start by adding and styling our toast. It's just some plain HTML next to the star bar, coupled with a CSS animation to make it disappear after a short time. The icons used are also from heroicons. I've used the check-circle and exclamation icons.

.toast {
  animation: toast 3s linear 1 both;
}


@keyframes toast {
  0% {
    opacity: 0.0;
    visibility: hidden;
  }


  3% {
    opacity: 1.0;
    visibility: visible;
  }


  80% {
    opacity: 1.0;
    visibility: visible;
  }


  100% {
    opacity: 0.0;
    visibility: hidden;
  }
}

By setting visibility: hidden we remove the toast from the accessibility tree and focus events, though it still affects layout. We use this over display: none since it can be used in an animation, we would need to use javascript to apply a change to the display attribute and it doesn't matter to us here that it still affects layout - you might argue that it's
better since nothing will move when the toast completes.

<!-- app/views/movies/show.html.erb -->

<!-- Within the same flex group as the stars
     but after all the stars have been rendered -->

<% if @toast %>
  <div class="flex items-center gap-x-1 px-4 py-2
    bg-black/75 rounded text-white text-sm
    relative top-[-2px]
    toast">
    <% if @toast == :success %>
      <%= inline_svg_tag("check.svg",
        class: "h-4 w-4 text-green-500 relative top-px") %>
      <span>Rating Added!</span>
    <% else %>
      <%= inline_svg_tag("warning.svg",
        class: "h-4 w-4 text-red-500 relative top-px") %>
      <span>Rating could not be added</span>
    <% end %>
  </div>
<% end %>

The @toast variable comes from the update action. This is why we chose to render the :show template, rather than redirecting. Though we could achieve the same effect with a redirect by making use of the flash object.

def update
  @movie = Movie.find(params[:id])

  if @movie.update(rating: params[:movie][:rating])
    @toast = :success
  else
    @toast = :warning
  end

  render :show
end

We simply set the @toast variable to whether or not the update happened successfully. Since the show action does not define the @toast variable, the message only shows in response to a call to the update action, i.e. clicking on a star.

Filling the Star Bar

If you've ever used a star bar in the past (I'm guessing you have) you'll have noticed that they tend to fill up from 0 to the point you hover over. This is an effect we can reproduce with a small amount of stimulus.

To start with, create a new stimulus controller by running this in your terminal. This will make sure your stimulus manifest is up to date (not applicable if you're using webpacker), as well as add the boilerplate for the controller and create a correctly named file.

bin/rails generate stimulus starBar

The basic expectation of the star bar is that when you hover over a star, that star and all the stars within an index lower than that star will show their hover state. When not hovering over any star, the star bar should simply display the rating.

The StarBarController looks like this.

import { Controller } from "@hotwired/stimulus"

export default class extends Controller {
  static targets = ["star"];
  static classes = ["hover"];

  enter(event) {
    this._fillToStar(event.params.starIndex);
  }

  leave() {
    this._fillToStar(-1);
  }

  _fillToStar(star) {
     this.starTargets.forEach((target, index) => {
      if (index <= star) {
        target.classList.add(this.hoverClass);
      } else {
        target.classList.remove(this.hoverClass);
      }
    });
  }
}

The two actions enter and leave simply toggle the hover class on the appropriate stars, depending on which star the user is hovering over or none if the user is not hovering over any stars.

The controller has one target called star. Since we expect a list of these (the 5 star icons) we're using this.starTargets in the plural to access them. The controller also has a class called hover - we use this so we can define the hover class for the stars in the markup rather than hardcode them in the controller.

Now we just need to add to markup to connect our star bar to the controller.

Our star bar now looks like this.

<div class="flex gap-x-3 mb-4"
  data-controller="star-bar"
  data-star-bar-hover-class="fill-yellow-500 stroke-yellow-500">
  <% Movie::MAX_RATING.times do |n| %>
    <%= button_to movie_path(@movie),
      method: :patch,
      params: { movie: { rating: n + 1 } } do %>
      <%= inline_svg_tag("star.svg",
        class: "w-8 h-8 #{star_rating_class(@movie, n)}",
        data: {
          star_bar_target: "star",
          star_bar_star_index_param: n,
          action: "pointerenter->star-bar#enter
            pointerleave->star-bar#leave"
        }) %>
    <% end %>
  <% end %>
  <!-- the toast is omitted, but would be here -->
</div>

The definition for the hover classes, data-star-bar-hover-class has to be on the same HTML element that attaches the controller. We also mark each of the stars as part of the star target with data-star-bar-target="star", we don't have to treat it differently just because it is going to be used as an array within the controller.

When the user hovers over a star we need to toggle the hover class on for each of the stars up to the star the user hovered over. We can pass parameters to our stimulus controller actions. In our case, we need to pass the index of the
star the user hovered over. This is what the data-star-bar-star-index-param="n" is doing. We can then access the parameter in the action with event.params.starIndex. The generic pattern for this is data-controller-name-param-name="value", the parameter can then be accessed from the action with event.params.paramName - the parameters name is
converted to snakeCase, just like the controller names are.

This star bar is very limited, I only really did it for some fun. For a more complete implementation of a star rating bar, check out a component library like Shoelace.

Conclusion

We've managed to create a fully-featured, interactive star rating bar and most of the changes were in our HTML. We didn't have to write any javascript if we didn't want to.

Making a Single Page Search with Hotwire

Tony Rowan — Wed, 13 Jul 2022 08:57:30 +0000

Cover photo by Tim Meyer on Unsplash

Turbo (part of the Hotwire suite) is a supercharged version of Turbolinks. If you're used to immediately turning off Turbolinks in any new project, you might be surprised to learn that you don't need React or Vue to build a rich and interactive web app.

What We're Building

We're going to be adding a single page search to a simple Single Page App (SPA) that allows users to view information about some Movies.

Starting Point

The starting point is heavily based on 'Using Hotwire with Rails for a SPA like experience' by Mike Wilson. If you want to follow along with this post you can use the starting point here on Github or follow along through that post. You should have a simple app where you can select a movie without reloading the page, with only a very simple rails controller.

class MoviesController < ApplicationController
  def index
    @movies = Movie.all
  end

  def show
    @movie = Movie.find(params[:id])
  end
end

Adding Search

The first step is to add search support to the controller.

class MoviesController < ApplicationController
  def index
    @movies = movies
    @query = query
  end

  def show
    @movie = Movie.find(params[:id])
  end

  private

  def movies
    if query
      Movie.where("title ILIKE ?", "%#{query}%")
    else
      Movie.all
    end
  end

  def query
    params[:query]
  end
end

We can verify this is working by simply adding the query parameter to the URL and visiting the page again. Only the movies matching the query you made are present in the list.

Not very user friendly though. Let's add a search form to the top of the list of movies, within the turbo-frame-tag named :index - the same turbo-frame with the list of movies.

<%= form_with(url: movies_path, method: :get) do |f| %>
  <%= f.text_field(:query, value: @query, placeholder: "Search") %>
<% end %>

Now when you make a search and hit enter, the list of movies is filtered. You'll also notice that, if you have a movie selected, it stays selected. Searching doesn't cause a full page refresh, only the list of movies updates.

This happens because each turbo-frame creates its own navigational context. By default, all links and form submissions from within a given turbo-frame only effect that turbo-frame. Under the hood, turbo is hijacking the submission event, making an AJAX call for the form submission and interpreting the result.

When the submission is successful turbo reads the response, looking for a turbo-frame-tag that has the same name as the turbo frame that caused the submission. If it finds a match, it swaps out the contents.

In our case, the form submission returns the same page so it is pretty clear that there will be a matching turbo-frame - the :index frame.

I said this was the default behaviour. By adding a turbo-target attribute to a form or link turbo will instead look for, and swap the contents in, the turbo frame with the name you passed - this is how clicking the movies changes the :details turbo frame and not the :index frame.

Come Alive with Stimulus

This is pretty good, but what if the user didn't have to press enter when they were done searching? We can search as the user types and we can do this with very little Javascript thanks to Stimulus - another part of the Hotwire suite.

We can watch for input changes on the text field of the search form and call an action on a Stimulus controller to submit the form for the user. This can be achieved with a small amount of markup on the form and the text field.

<%= form_with(
  url: movies_path,
  method: :get,
  data: {
    controller: "submit-form",
    submit_form_target: "form",
  }
) do |f| %>
  <%= f.text_field(
    :query,
    value: @query,
    placeholder: "Search",
    data: {
      action: "input->submit-form#submit"
    }
  ) %>
<% end %>

On the text field we've added data-action="input->submit-form#submit". The value assigned to the data-action attribute encodes which event to listen for and which controller and action (method) to invoke when the event happens. The generic markup is event->controller-name#action. In our example, event is input, controller is submit-form and our action is submit.

The input event name comes from the JS DOM events, and we want to listen to any changes in the input. The controller name submit-form is just the name we gave the controller we want to invoke here - it is expected to be defined in a file named submit_form_controller.js. Its name in Javascript land will then be SubmitFormController. The submit action is just a public method defined on that controller.

On the form element we've added data-controller="submit-form" and data-submit-form-target="form". The former simply tells Stimulus to attach a new instance of the SubmitFormController to the form element, allowing us to invoke its actions from the form element or any of its children. The latter attribute makes the form element available from the controller as a 'target' called form - from the controller we will be able to access the element by calling this.formTarget.

The SubmitFormController is very simple.

import { Controller } from "stimulus";

export default class extends Controller {
  static targets = ["form"];

  submit() {
    clearTimeout(this.timeout);
    this.timeout = setTimeout(() => {
      this._submit();
    }, 300);
  }

  _submit() {
    this.formTarget.requestSubmit();
  }
}

When the submit action is called it submits the target, which is assumed to be a form. The action itself is debounced. This stops us from generating too many simultaneous requests which makes the submissions overlap.

The this.formTarget.requestSubmit(); is very important, if we used submit() rather than requestSubmit(), the event that Turbo hooks into the intercept the form submission would not happen and you would have a normal form submission - with a full page refresh.

A Small Problem

This works really well - until the first time the form is submitted - then the focus in the text field is lost and we can't keep typing. That's definitely a bug!

What's happening here is that when the response is returned, we're replacing the whole contents of the :index turbo frame - including the search form itself. To avoid this we can move the search form outside of the turbo frame, and then add turbo-target: :index to the form element. This will mean when the form is submitted, turbo will replace the contents of the :index turbo-frame and leave the search field completely untouched.

Conclusion

And there we have it. We've added a single page search to an app and all we needed was a small amount of markup and very small amount of Javascript. That's the power of Turbo.

The resulting code can be found on Github.

Weekend Project: Pokémon Name Generator

Tony Rowan — Tue, 12 Jul 2022 14:56:40 +0000

Cover Photo by Jon Tyson on Unsplash

Like many millenials, I grew playing/watching/collecting Pokémon. Back then, there were only 151, but that was still enough variety to make naming all of them quite difficult - even if you were fan. For parents and the uninterested, the names were quite difficult to keep in one's head. Now there's almost 1000 and some of them are quite strange. Looking at you "klink, klang, and klinklang".

Ah yes, pokechu! The electric mouse Pokémon!

With so many Pokémon now, I thought it would be fun to see if I could generate names that sounded like real Pokémon using ruby.

Naïve Attempt

My first thought was to simply create names of roughly the right length with roughly the right letters. Simply gather these stats from the corpus of actual Pokémon names and then generate names that match these stats.

statistics = {
  letter_count_distribution: names.map(&:size)
  letter_distribution: names.split("").flatten
}

name = statistics[:letter_count_distribution].sample.times.map do
  statistics[:letter_distribution].sample
end.join

puts name

That didn't work. It produced unpronouncable nonsense.

eareredneemoo
gedgfa
lbphoobng
ureerbeier
airmr

The problem was that this naïve algorithm doesn't take into account letter to letter changes. The letters lbph can't all appear next to each other.

That sounds like what a Markov Chain is for!

Markov Chains

I'm not an AI or language processing expert, but I vaguely remember covering Markov Chains at uni. Their main use is predication. For instance, predicting which word you mean, given the first few letters you've typed.

I wasn't trying to predict which name the user was typing as they were typing it - I was trying to create new names. So I had to use the concepts of a Markov Chain a little loosely.

At the core of Markov Chains is the endcoding of all the valid ways to get from one letter (or phoneme or word or w/e) to the next, including which letters can start the name and which names can end it. That's exactly what was missing from the naïve algorithm.

So instead of simply gathering stats on length and letter frequency, I built up map of valid letter transitions and their relative frequencies. From this map all I had to do was choose a random letter to start, and then random subsequent letters until the name ended when it selected nil.

# create a statistics map of all the valid transitions
markov_statistics = {}

pokemon_names.each do |name|
  last_letter = nil

  name.split("").each do |letter|
    if markov_statistics[last_letter]
      markov_statistics[last_letter] << letter
    else
      markov_statistics[last_letter] = [letter]
    end

    last_letter = letter
  end

  if markov_statistics[last_letter]
    markov_statistics[last_letter] << nil
  else
    markov_statistics[last_letter] = [nil]
  end
end

# create the name by picking valid letters
pokemon_name = ""
current_letter = statistics[nil].sample

loop do
  break if current_letter.nil?

  pokemon_name << current_letter
  current_letter = statistics[current_letter].sample
end

puts pokemon_name

The markov_statistics is a hash that encodes all the valid transitions in the corpus. The key of the hash is the start letter and the values of the hash are all the letters that might follow the start letter. For letters that ended the
name nil is included in the value array.

For example:

corpus = ["bulbasaur", "charmander", "squirtle"]

statistics = {
  nil => ["b", "c", "s"],
  "a" => ["r", "s", "u"],
  "b" => ["a", "b"],
  ...
}

I allowed letters to be included in the value array multiple times so that when I choose a letter from the value array with .sample it was weighted to the frequency of that pair in the corpus. That way more common pairs would be chosen more often.

This produced better names, but some of them only had one letter in them. How is that valid?

s
munegogunchezatoy
pitulor
kiterlving
higraponeo

Context

The problem was that nil->s was valid, and so was s->nil. So, the algorithm could choose nil->s->nil - it thinks s is a perfectly valid name!

This fix for this was to introduce some context. In markov chains, we consider all the letters typed so far. I didn't want to consider all the letters in the name before choosing the next letter - then it would only produce actual Pokémon names! But, I could consider more that one letter at a time.

# create statistics of valid transitions,
# given a configurable length of context
markov_statistics = {}

pokemon_names.each do |name|
  context = []

  name.split("").each do |letter|
    if markov_statistics[context]
      markov_statistics[context] << letter
    else
      markov_statistics[context] = [letter]
    end

    context = [context, letter].flatten
    context = context.drop(1) if context.size > context_length
  end

  if markov_statistics[context]
    markov_statistics[context] << nil
  else
    markov_statistics[context] = [nil]
  end
end

# create the name by picking valid letters,
# keeping a configurable length of context
pokemon_name = ""
context = []
current_letter = statistics[context].sample

loop do
  break if current_letter.nil?

  pokemon_name << current_letter
  context << current_letter
  context = context.drop(1) if context.size > context_length
  current_letter = statistics[context].sample
end

puts pokemon_name

In this version markov_statistics contains a map where the keys are a list of letters that form some context and the values are the possible next letters e.g. [b, u, l]->[b, g, ...] where bul can be followed by b or g etc.

After this, and after playing around with a sensible value for the length of the context, I got some better sounding names.

turne
vespichu
ursarill

As the size of the context increased, it produced actual Pokémon names with greater and greater frequency. Above 5 all the names produced were actual Pokémon, in machine learning this is called over-training.

Comparing

With testing real AI/ML applications, the system is given half of the training or classification data, and the other half is used to test the classification/prediction accuracy. I could do that with my generator.

If I gave the generator half the available Pokémon names, could it generate the other half?

actual_pokemon = corpus.pokemon.shuffle

mid_point = (actual_pokemon.size + 1) / 2
training_data = actual_pokemon[..mid_point]
test_data = actual_pokemon[mid_point..]

algorithm = Markov.new(training_data, context_length: 3)
generated_names = 5000.times.map { algorithm.generate_name }.uniq
test_data_names_generated = test_data
  .map { |datum| datum.join("") }
  .intersection(generated_names)

puts "Test Data Names Generated: #{test_data_names_generated}}"

Well, the answer was no. It produced roughly 3 of the test data names for every 5000 names produced. As a classification/prediction engine that would be pretty poor performance. But for a fake name generator, that's not bad.
It actually produces names like "chespin" and "porygon" without ever even seeing them before. I call that a win.

Is it a Pokémon?

Can it trick a person? That was the whole point of this after all. To answer that question, I built a quiz app to do just that.

I produced 10,000 names from the generator. This includes some real and some not real names. The app randomly picks one and which you think it is.

Try it now!

Phonemes

Another extension I wanted to try was to consider phonemes as the base unit of the names, rather than just letters. For this I needed to collect a list of phonemes in the English languge and then break the names into groups of letters based on those phonemes.

pokemon_phonemes = pokemon_names.map do |this_pokemon_name|
  remaining_pokemon_name = this_pokemon_name
  this_pokemon_phonemes = []

  loop do
    break if remaining_pokemon_name.empty?

    phoneme = phonemes.find { |phoneme| remaining_pokemon_name.start_with?(phoneme) }

    this_pokemon_phonemes << phoneme
    remaining_pokemon_name = remaining_pokemon_name.sub(phoneme, "")
  end

  [this_pokemon_name, this_pokemon_phonemes]
end.to_h

The resulting hash had entries that looked like this.

{ "charmander" => ["ch", "ar", "m", "a", "n", "d", "er"] }

The hope I had with this extension was to be able to treat certain groups of letters that make a sound (like "ch") as if it was a single letter in the Markov Chain. I hoped this, in turn, would restrict the possibilities for each phoneme, creating more realistic suggestions.

It didn't really make too much of a difference, though.

Code

You can find the code, both for the name generator and the quiz on Github.

DEV Community: Tony Rowan

Mock User Agent in View Component Specs

How To: Implement an Android Smart App Banner

So, How Do You Implement It?

Example

I Don't See the Prompt

Bypass User Engagement Checks

Insecure Origins Treated As Secure

Advanced Usage

Conclusion

A Deep Dive into Memory Leaks in Ruby

Finding Leaks in Ruby

memory_profiler in Ruby

derailed_benchmarks in Ruby

perf:mem_over_time

perf:objects

perf:heap_diff

Leaks in Real Ruby and Rails Apps

Circular References

Application Performance Monitoring: The Event Timeline and Allocated Objects Graph

Wrapping Up

How to Track Down Memory Leaks in Ruby

Ruby Memory Management

Garbage Collection in Ruby

What Does A Memory Leak Look Like in Ruby?

Find Ruby Leaks with the Garbage Collector Module

Application Performance Monitoring in Production with AppSignal

Wrap Up and Further Reading

Adding a Star Rating with Hotwire

What We're Building

Add Ratings

Add the Star Bar

Submit a Rating

User Feedback

Filling the Star Bar

Conclusion

Making a Single Page Search with Hotwire

What We're Building

Starting Point

Adding Search

Come Alive with Stimulus

A Small Problem

Conclusion

Weekend Project: Pokémon Name Generator

Naïve Attempt

Markov Chains

Context

Comparing

Is it a Pokémon?

Phonemes

Code

`memory_profiler` in Ruby

`derailed_benchmarks` in Ruby

`perf:mem_over_time`

`perf:objects`

`perf:heap_diff`