DEV Community: Gabriel Nagy

Faster module tests with Facter 4 and rspec-puppet

Gabriel Nagy — Sat, 22 Jan 2022 10:03:16 +0000

The latest major version of Facter—Puppet's tool for collecting system information—has been out for some time now. However, we've been hard at work fixing bugs ever since. The fact that Facter has to be able to run on a variety of operating systems and architectures makes maintenance quite a challenging ordeal.

Returning to using Ruby as the language of choice (Facter 3 was written in C++) meant that we'd have a lot more freedom in structuring and writing the code, mostly at the expense of the added run time. This isn't an issue for how most people use Facter, for example Puppet loads all available facts at once when it runs, so a few seconds of added run time won't make a difference with Puppet runs that already take minutes. This was something we took into account when making the decision to return to Ruby with Facter 4 (initially Facter was written in Ruby, but was then rewritten in C++ for improved performance). What we didn't take into account were the intricacies of how Facter interacts with other downstream projects that we weren't aware of, such as the rspec-puppet test framework.

What is `rspec-puppet`?

Well, RSpec is a well-known test framework for Ruby, and Puppet is... well, you probably already know or you wouldn't be here reading these words.

rspec-puppet is the tool of choice when writing unit tests for Puppet modules. It provides a helpful syntax for interacting with Puppet catalogs in an RSpec way. Since it's unfeasible to acceptance test Puppet functionality on dozens of operating systems and versions, rspec-puppet circumvents this by making Facter trick Puppet into thinking it runs on different OS configurations.

Puppet gets almost all of its system-related information using facts, so for example if you're running some Linux and you're curious to see how your Puppet manifest would behave on macOS, in most cases it's enough to feed Facter a bunch of macOS facts. Of course, the underlying implementation is more complicated than that, especially for Windows, but in a nutshell this is how rspec-puppet works.

For each test, rspec-puppet stubs the fake facts using the custom facts API. The fake facts it gets from facterdb, which is a gem that contains "dummy" facts for a variety of operating systems and Facter versions—basically lots of files containing facter --json output. The resulting information is then fed to Puppet for catalog compilation.

Getting slower and slower...

Over time, people have been noticing that module tests running with Facter 4 were much slower than Facter 2. Module tests have skipped Facter 3 altogether because even though Facter 3 ships a compatible Ruby module, in the end it's C++ code that's extremely incompatible with the Ruby ecosystem (you can't gem install it like you would any other gem).

After some investigation it turned out that Facter 4 was evaluating underlying core facts even though they had been overridden by custom facts. The first instinct would be to classify this as a bug, but a closer look showed us that this functionality was intended in order to be fully-compatible with Facter 3. And because modules never ran tests with Facter 3 this wasn't a problem until now.

Overriding a core fact, similar to how rspec-puppet does:

Facter.add(:ipaddress, weight: 999) do
  setcode { '1.1.1.1' }
end

will return the correct value (1.1.1.1) but it will cause Facter to also load the core ipaddress fact, ultimately resolving all networking facts, which means executing system commands.

Assuming our custom ipaddress fact is defined inside the custom_facts directory, here's what gets called when we attempt to resolve the fact:

$ FACTERLIB=$PWD/custom_facts strace -f -eexecve facter ipaddress
[pid 1777063] execve("/usr/bin/ip", ["ip", "-o", "link", "show"], 0x563bf6b7acf0 /* 110 vars */) = 0
[pid 1777067] execve("/usr/bin/ip", ["ip", "link", "show"], 0x563bf64eafa0 /* 110 vars */) = 0
[pid 1777068] execve("/usr/bin/dhcpcd", ["/usr/bin/dhcpcd", "-U", "lo"], 0x563bf6aa57b0 /* 110 vars */) = 0
# ... dhcpcd is called for every network interface on the system
# I removed the other calls for brevity

[pid 1777078] execve("/usr/bin/ip", ["ip", "route", "show"], 0x563bf6aba230 /* 110 vars */) = 0
[pid 1777079] execve("/usr/bin/ip", ["ip", "-6", "route", "show"], 0x563bf6ae37d0 /* 110 vars */) = 0

Doing this hundreds or even thousands of times per test suite definitely adds to the increase in run time, and it's redundant since the resolved facts are never needed.

Fixing things without breaking more things

Modifying Facter's behavior—undocumented as it was—was a no-go from the start, as we found out before (Facter is a good example of Hyrum's law in action). So, we had to think of other ways to improve performance.

We started by decoupling Puppet from Facter as much as we could, introducing the possibility of having multiple Facter backends. While Puppet would use the default Facter implementation when running on its own, external users would be able to define and pass their own Facter implementation when initializing Puppet, similar to how puppetserver configures Puppet to use its JRuby-compliant HTTP client.

To avoid breaking the Facter API, we ended up implementing an overcomplicated way of interacting with a hash. Using our dumb Facter backend, custom facts were now simply added to a hash, and querying them would just produce them from the hash if available:

class FacterTestImpl
  def initialize
    @facts = {}
  end

  def value(fact_name)
    @facts[fact_name.to_s]
  end

  def add(name, options = {}, &block)
    raise 'Facter.add expects a block' unless block_given?
    @facts[name.to_s] = instance_eval(&block)
  end
  ...
end

With our custom implementation we bypassed Facter altogether. This managed to bring us back to Facter 2 speeds, which behaved similarly by just returning the custom fact's value without resolving any additional facts.

Of course, there may be downsides to this approach, as Facter code paths will no longer be executed by rspec-puppet. In the past there have been occasions where we merged Facter work that passed our CI, but ended up failing in module tests, so switching to this implementation will get rid of this level of testing. I'd argue that it wasn't a module's business to validate Facter itself, but it was a good safety net for us as maintainers.

And because performance improvements mean nothing without showing the numbers, here's how test times have changed for the puppet-nginx module:

Running rake parallel_spec on the module using Puppet 7 / Facter 4 took around 47 minutes with the original rspec-puppet implementation.

We managed to shave off around 11 minutes off the test run by using the custom Facter implementation in rspec-puppet.

One thing I haven't mentioned is that running the same tests with Puppet 6 takes a total of 25 minutes, so there's more to improve in Puppet itself as well. However, from a Facter standpoint it's impossible to make the tests any faster, unless Ruby itself improves hash access speed 😜.

This new functionality is opt-in and configurable by setting the facter_implementation RSpec option in your spec_helper.rb.

RSpec.configure do |config|
  config.facter_implementation = :rspec
end

It was first made available in rspec-puppet 2.11.0, with an additional bugfix that was released in rspec-puppet 2.11.1.

The investigation and work surrounding this improvement has spanned many months and was the product of multiple Puppet employees and community members, namely Gimmy, Josh Cooper, Ewoud Kohl van Wijngaarden and Tim Meusel. Thanks to everyone who contributed!

Windows, Ruby and Long Paths

Gabriel Nagy — Mon, 07 Jun 2021 12:58:30 +0000

Over the past few months, we've had a long-standing issue related to Puppet on Windows resurface (puppetlabs/Puppet.Dsc#144). More specifically, Puppet modules with long file paths could not be installed on Windows due to a limitation in the Windows operating system. In short, if a file path surpasses 260 characters, it's open season: the path has to be referred to in a different format, first-party apps like Notepad or File Explorer start to behave erratically, and there's no guarantee what works and what doesn't anymore.

Buckle up, as we're about to go on a perilous journey where we'll encounter and modify old code, graft resources onto Windows executables, build Ruby with the help of renowned triple-A videogame Hitman^TM (I'm 100% serious) and generally have a good time.

Part 1: The Windows

In theory, Windows's NTFS filesystem supports a maximum of an approximate 32767 characters in a file path. However, there's also a hard limit of 260 characters, MAX_PATH, which is enforced in all Win32 API file management functions.

A short Google search for "windows long file paths" shows that this issue is frequently hit by developers and regular users alike. Since 260 characters is really not that much for a file path, the limit can be hit easily. Off the top of my head I remember seeing the error when installing the boost libraries on Windows, fortunately I was able to work around it by specifying a shorter install path.

Bypassing the `MAX_PATH` limitation

If there's one thing I've grown to appreciate from Microsoft, it's their extensive API documentation (I'm looking at you, Apple 👀). For this long path problem they've put together a nice document¹ detailing how to work around MAX_PATH.

The two options are as follows:

specify long paths using the extended-length format (e.g. \\?\D:\very long path)
- this format does not support relative paths
- it would also require extensive refactoring throughout any software that decides to implement long path support
disable the limitation by changing a registry value (needs at least Windows 10, version 1607)
- doing this will remove the limitation in the Win32 functions, and will enable them to work with long paths without the extended prefix

The second options comes with a catch, which we failed to notice when we first investigated the problem. In Microsoft's article, just under registry example lies an application manifest:

<application xmlns="urn:schemas-microsoft-com:asm.v3">
    <windowsSettings xmlns:ws2="http://schemas.microsoft.com/SMI/2016/WindowsSettings">
        <ws2:longPathAware>true</ws2:longPathAware>
    </windowsSettings>
</application>

What we thought needs to happen: by conflating other (wrong) sources with Microsoft's official documentation, we came to the (wrong) conclusion that the MAX_PATH limitation is globally controlled by the registry key, and on a per-application basis through the application manifest. This. Is. Not. True.

What actually needs to happen: in addition to having the registry key set, the application that wants to use long paths NEEDS to embed that application manifest at build time. Microsoft devs are playing it extra safe here, showing how much they care about backwards compatibility. And in a way they're right, who knows how much of the third-party software out there makes wrong assumptions about this limitation so it would only make sense for it to be an opt-in feature.

This definitely explained why most of the things did not Just Work^TM after setting the registry key, but it also gave us hope. Well, we still needed to figure out what the hell an application manifest was, how to embed it in the executable, and if possible to automate all this without the help of the Visual Studio GUI.

Application Manifests! What do they know? Do they know things?? Let's find out!

"When in doubt, check what the Python folks do." -anonymous proverb

Quoting from the Windows documentation, application manifests are XML files that describe and identify the shared and private side-by-side assemblies that an application should bind to at run time.² Got it? Me neither, and this was as far as I was willing to go reading documentation 😆. I left it at "put manifest in executable file" for simplicity.

During my time at Puppet I've grown to be wary with everything regarding Windows development. It couldn't possibly be that hammering an XML file in an executable would magically get rid of the long path limitation (It does). There must be more to it (There wasn't). Other, more complicated theories were running through my head, among them a far-fetched one suggesting that the manifest was somehow read by Microsoft's compiler which in turn optimized the Win32 function calls, making this solution impossible to use with different compilers. I also presumed that this manfest thing is C# specific, hence not applicable to Ruby which is written in C.

From the comments in the related Ruby bug³ I knew that Python had this feature, so I started digging through their source code fully expecting to see path manipulation with the \\?\ prefix for every Win32 API call. Adding to the fact that the Ruby issue was like 5 years old at this point, I was sure that this long paths fix would be a massive effort and likely impossible without knowing the ins and outs of the Ruby C implementation.

I started going through the Python codebase, but all I could find was the addition of the longPathAware manifest key, which kind of dismantled all my preconceptions about how application manifests work. Maybe having the manifest was indeed enough.

From another useful Microsoft document,⁴ I found that a manifest can be embedded into an executable using the following syntax:

mt.exe -manifest MyApp.exe.manifest -outputresource:MyApp.exe;1

To do this for a library, replace the 1 at the end with 2.

As a quick sanity check, I opened up Visual Studio, created a new C project in which I simply called CreateDirectory with a long path. The call errored out as the path was too long, but after including the manifest it worked. Granted, it took me way too long to find out how to include a manifest through the VS user interface, but hey, it worked!

Part 2: The Ruby

Coming from a Linux background, I definitely did not expect to have so much fun compiling Ruby on Windows (it's been a few weeks since I've done this so the bad memories have mostly gone away).

Building this thing

Ruby on Windows can be compiled with both MinGW/GCC and Visual C++, and I decided to start with the latter as I already had the Visual C++ compiler installed.

Building Ruby from the git source requires an extra set of commands like bison, patch and sed. To get these on Windows I installed Cygwin and cyg-get, and made sure to have the Cygwin bin path properly set. After that it was just a matter of calling cyg-get to install each required package.

Afterwards, Ruby can be built by executing the following commands:

win32\configure.bat
nmake

Unfortunately my first Ruby build failed fast with a ucrtbase.dll error somewhere here. I still have no idea what this code does, I assume it searches for a function in my ucrtbase.dll. I blamed it on the fact that I'm running Windows builds from the Dev Channel which may have newer versions of DLLs, and I started my search for the perfect ucrtbase.dll. This is a good moment to plug Everything, an awesome search tool for Windows:

I kid you not, I ended up taking the ucrtbase.dll from my Hitman installation, copied it to the Ruby directory and prepended the path to the LIB environment variable. Thanks to Agent 47 I was able to successfully build Ruby with Visual C++.

Look at that unsettling smile! Also, did you know that Agent 47 is partly Romanian?

Where to put the manifest: The Visual C++ version

The good part is that I found a spot in the Makefile where mt.exe is called. The bad part is that mt.exe is a tool only provided in the Visual C++ toolchain, and the Makefile was Visual C++-specific, so this would only fix half of the problem. At Puppet, we vendor our own Ruby, but we compile it with MinGW/GCC so we wouldn't be able to benefit from the Visual C++ changes.

Either way, it was late at night and I just wanted to get that manifest inside the Ruby executable, so I started desecrating the Makefile to make it behave the way I wanted. I ended up embedding the manifest into EVERY executable and library generated by the compiler (including all native extensions), so I might have gone a bit overboard with that. On the bright side, I was able to confirm that long paths now worked!

Still, it was a piece of ugly code, I shared it in a comment on the original Ruby ticket, and to my surprise just a few hours later nobu responded with a cleaner solution which I validated, and looked something like this:

diff --git a/win32/Makefile.sub b/win32/Makefile.sub
index c88ae6f9d1..22198aa358 100644
--- a/win32/Makefile.sub
+++ b/win32/Makefile.sub
@@ -305,9 +305,10 @@ XCFLAGS = -DRUBY_EXPORT $(INCFLAGS) $(XCFLAGS)
 !if $(MSC_VER) >= 1400
 # Prevents VC++ 2005 (cl ver 14) warnings
 MANIFESTTOOL = mt -nologo
-LDSHARED_0 = @if exist $(@).manifest $(MINIRUBY) -run -e wait_writable -- -n 10 $@
-LDSHARED_1 = @if exist $(@).manifest $(MANIFESTTOOL) -manifest $(@).manifest -outputresource:$(@);2
-LDSHARED_2 = @if exist $(@).manifest @$(RM) $(@:/=\).manifest
+LDSHARED_0 = $(Q)$(MINIRUBY) -run -e wait_writable -- -n 10 $@
+LDSHARED_1 = $(Q)if exist $(@).manifest (set MANIFEST=$(@).manifest) else (set MANIFEST=$(win_srcdir)/ruby.manifest) && \
+            call $(MANIFESTTOOL) -manifest ^%MANIFEST% -outputresource:$(@);2
+LDSHARED_2 = $(Q)@$(RM) $(@:/=\).manifest
 !endif
 CPPFLAGS = $(DEFS) $(ARCHDEFS) $(CPPFLAGS)
 !if "$(USE_RUBYGEMS)" == "no"

If you look closely you can see that mt is called with 2 which means it only works for libraries. After some debugging I extended the Makefile with additional commands to make it work for executables as well, but the changes got a bit more complicated than I wanted, so I'll skip over them; especially since the final fix is completely different.

The remaining issue was to make it also work with GCC.

Where to put the manifest: The GCC version

After some Google searching I found out that the MinGW toolchain provides windres, a tool that can manipulate Windows resources. What are Windows resources you might ask? Well, various things that can be embedded into an application, like icons, cursors, fonts, and... application manifests!

I was able to find usage of windres inside the Ruby Cygwin Makefile:

%.res.@OBJEXT@: %.rc
    $(ECHO) compiling $@
    $(Q) $(WINDRES) --include-dir . --include-dir $(<D) --include-dir $(srcdir)/win32 $< $@

%.rc: $(RBCONFIG) $(srcdir)/revision.h $(srcdir)/win32/resource.rb
    $(ECHO) generating $@
    $(Q) $(MINIRUBY) $(srcdir)/win32/resource.rb \
      -ruby_name=$(RUBY_INSTALL_NAME) -rubyw_name=$(RUBYW_INSTALL_NAME) \
      -so_name=$(DLL_BASE_NAME) -output=$(*F) \
      . $(icondirs) $(srcdir)/win32

In Makefile lingo this means that .rc files are turned into .res files, and .rc files are created through the execution of the win32/resource.rb Ruby script. In short, for each Ruby executable and library generated by the compiler—ruby.exe, rubyw.exe, and the Ruby DLL library)—the script creates a .rc file containing various things like the Ruby icon and copyright information. After a quick look through the script, I found the place where the manifest can be included, and it was as simple as including the following line in the generated .rc file, provided that ruby.manifest contains the appropriate long path manifest:

1 RT_MANIFEST ruby.manifest

1 stands for the resource ID, RT_MANIFEST is the type defined in winuser.h for application manifests (it maps to the integer 24, which can also be used if you don't have access to the header file), and ruby.manifest is the file which contains the application manifest.

Below is a simplified version of the win32/resource.rb code that generates the .rc files, with the newly added manifest line:

[ # base name    extension         file type  desc, icons
  [$ruby_name,   CONFIG["EXEEXT"], 'VFT_APP', 'CUI', ruby_icon],
  [$rubyw_name,  CONFIG["EXEEXT"], 'VFT_APP', 'GUI', rubyw_icon || ruby_icon],
  [$so_name,     '.dll',           'VFT_DLL', 'DLL', dll_icons.join],
].each do |base, ext, type, desc, icon|
  next if $output and $output != base
  open(base + '.rc', "w") { |f|
    f.binmode if /mingw/ =~ RUBY_PLATFORM

    f.print <<EOF
#include <windows.h>
#include <winver.h>

#{icon || ''}
1 RT_MANIFEST ruby.manifest
VS_VERSION_INFO VERSIONINFO
 FILEVERSION    #{nversion}
 PRODUCTVERSION #{nversion}
 FILEFLAGSMASK  0x3fL
 FILEFLAGS      0x0L
 FILEOS         VOS__WINDOWS32
 FILETYPE       #{type}
 FILESUBTYPE    VFT2_UNKNOWN
BEGIN
 BLOCK "StringFileInfo"
 BEGIN
  BLOCK "000004b0"
  BEGIN
   VALUE "Comments",         "#{RUBY_RELEASE_DATE}\\0"
   VALUE "CompanyName",      "http://www.ruby-lang.org/\\0"
   VALUE "FileDescription",  "Ruby interpreter (#{desc}) #{sversion} [#{RUBY_PLATFORM}]\\0"
   VALUE "FileVersion",      "#{sversion}\\0"
   VALUE "InternalName",     "#{base + ext}\\0"
   VALUE "LegalCopyright",   "Copyright (C) 1993-#{RUBY_RELEASE_DATE[/\d+/]} Yukihiro Matsumoto\\0"
   VALUE "OriginalFilename", "#{base + ext}\\0"
   VALUE "ProductName",      "Ruby interpreter #{sversion} [#{RUBY_PLATFORM}]\\0"
   VALUE "ProductVersion",   "#{sversion}\\0"
  END
 END
 BLOCK "VarFileInfo"
 BEGIN
  VALUE "Translation", 0x0, 0x4b0
 END
END
EOF
  }
end

A tool that helped me a lot in debugging the executables generated by GCC and Visual C++ is Resource Hacker. It can open up executables and show you what resources they contain. This was how I was able to notice that if I set an ID different than 1 to the manifest, GCC would include a default manifest which shadowed my long path manifest, causing the feature to no longer work.

Right above the manifest we just added, there's also the Version Info resource which we saw in the code above!

I glossed over the build process for MinGW/GCC, because it's... not as complicated and it didn't involve any Hitman DLLs. I did it using the MSYS2 toolchain which gives you a bash prompt, then compiled Ruby as if I was on Linux.

Conclusions

After some digging I realized that the resource.rb script which created the .rc files with the manifest was also executed during the Visual C++ build, so changing that Ruby script would accomodate both compilers without the need for additional code changes. One thing to note is that with Visual C++, the rc tool is used instead of windres which achieves similar results, and there's no need for mt anymore.

I hurried to open a pull request, where nobu again provided feedback and promptly merged it!

In the end it was one of those few-line fixes with a huge impact. I'm happy it could be done in a few lines of code, and that I had the chance to learn a lot about Windows and Ruby on Windows in the meantime.

Things I learned by working on this:

If something fails to compile, copying random DLLs around might just fix your problems
Make sure the code you expect to run is actually running
- if you modify something like a Makefile and nothing appears to change, don't blame your coding skills just yet—find a way to figure out if and when that code path is executed (nmake V=1 may provide more context for Ruby on Windows)
Isolate the problem you want to fix
- when I was unsure about what the application manifest did, I validated it with the shortest possible C program to confirm its behavior; this way I knew what I was going for when looking through the Ruby codebase
Find a reliable way to validate your changes
- for this type of problem, the solution consisted in making the OS aware of the fact that it should enable long paths support, so it's helpful to figure out how that actually happens for an application; i.e. getting as close as possible to how Windows makes this check
- at first I tested my changes by creating directories with paths longer than 260 characters through irb, but how do you dig deeper when that doesn't work?
- after a lot of trial and error, I found that the most reliable way to validate my changes was to open ruby.exe in Resource Hacker, and make sure it only included my manifest, with ID 1
Sleep on it, and don't be afraid to experiment
- I managed to get a working fix in the first day, but with each following day I changed up the code, and the final fix ended up being in a totally different place than initially expected
- when you feel that something gets more and more complicated and you're not even close to fixing the problem, see if you can approach it in a different way; this made for a way cleaner solution in my case

To finish things up, here's an oversimplified diagram of the Ruby on Windows build process:

And here's the story of Ruby trying to access long paths on Windows:

The End?

If you take a closer look at the GitHub issue I referred to in the introduction, this does not fully solve our problem in Puppet. On Windows, we use the minitar gem to build and install Puppet modules (which are just tarballs downloaded from the Puppet Forge). Unfortunately, there is an issue with minitar being unable to unpack the tarballs it creates if the file paths exceed a certain length. This leaves us in a state where the modules in question can be built, but cannot be installed.
The plan is to attempt to fix the issue in minitar, and maybe blog about it, so stay tuned for the next installment in the Long Paths series!