DEV Community: Martin Alfke

Das neue Puppet Open Source Projekt heißt "OpenVox"

Martin Alfke — Tue, 28 Jan 2025 15:00:38 +0000

Puppet by Perforce hat angekündigt,
dass die Open Source Pakete ab 2025 nur noch nach Zustimmung einer nicht weiter definierten EULA verfügbar gemacht werden sollen.

In early 2025, Puppet will begin to ship any new binaries and packages developed by our team to a private, hardened, and controlled location. Community contributors will have free access to this private repo under the terms of an End-User License Agreement (EULA) for development use.

Dabei muss berücksichtigt werden, dass die kostenfreie Variante nur für maximal 25 Nodes gilt.
Wer mehr Systeme mit Puppet Open Source verwalten will, benötigt eine Puppet Labs Support Commercial License.

Die Open Source Community hat das Gespräch mit dem Puppet Management gesucht.

Hier das Resultat:

Das neue Puppet Open Source Projekt heißt OpenVox
- Vorbereitung
- Aus Puppet wird OpenVox und Puppet Open Source
- Source Code, Pakete und Container
- Ich nutze Puppet Open Source. Was muss ich tun?
- Puppet Open Source
- OpenVox
- Die weitere Entwicklung
- Wie kann man helfen?
- Professioneller Support
- Zusammenfassung

Vorbereitung

Vor dem ersten Gespräch mit Puppet hat sich die Community im Vorfeld besprochen und analysiert, ob man als Open Source Projekt in der Lage sein kann, die weitere Entwicklung zu stemmen.
Das Feedback aus der Open Source Community und von Open Source Usern war sehr positiv.

Einige Nutzer haben sogar direkt angekündigt, dass man Entwickler für dedizierte Aufgaben zur Verfügung stellen möchte.

Aus Puppet wird OpenVox und Puppet Open Source

Die Open Source Community hat sich mit Puppet by Perforce darauf geeinigt, dass die weitere Entwicklung von Puppet in einem Fork unter der Verwaltung der Open Source Community erfolgen soll.

Nach einer öffentlichen Abstimmung Ende 2024 innerhalb der Community, hat man sich für den Namen OpenVox entschieden und beschlossen, dass die Entwicklung innerhalb der bereits existierenden Vox Pupuli Puppet Open Source Community auf GitHub stattfinden soll.

Da Perforce die Namensrechte an Puppet besitzt, hat man gemeinsam mit Perforce entschieden, dass die Pakete, die durch das OpenVox Projekt erstellt werden, nicht mehr Puppet im Namen haben sollen.
Dies ist auch in Hinsicht auf Differenzierbarkeit der Herkunft der Pakete sinnvoll.
Die Open Source Pakete von Perforce werden weiterhin puppet-agent, puppetdb und puppetserver heißen.

Gemeinsam mit Perforce wurde vereinbart, dass man sowohl die Kommandozeile, wie auch Namen der Konfigurationsdateien und die Schnittstellen zur Erweiterung von Puppet ( /lib/puppet in Modulen) zu existierenden Installation und Code kompatibel halten möchte.

Die weitere Kompatibilität soll durch ein Language Steering Committee sichergestellt werden, in dem alle Beteiligten gemeinsam über Änderungen an der Puppet DSL oder Inkompatibilitäten entscheiden.

Source Code, Pakete und Container

Der frei verfügbare Puppet Open Source Code wurde in das OpenVox GitHub Projekt geforkt.

Im ersten Schritt hat das Re-branding stattgefunden.
Für Anwender ist es wichtig zu wissen, dass Kommandozeilen Tools, Bibliotheken und Puppet Erweiterungen auch unter OpenVox vollständig kompatibel zu Puppet Open Source sind.

Aktuell werden die Pipelines zur Erstellung der RedHat und Debian Pakete auf öffentliche GitHub Actions umgestellt und sollen sehr bald zur Verfügung gestellt werden.
Erste reproduzierbare Build wurden bereits getetstet.
Pakete für Windows und MacOS sind ebenfalls geplant.

Die Puppet Container (puppetserver und puppetdb) werden nicht mehr weiter gepflegt!

Statt dessen werden OpenVox Container zur Verfügung gestellt. Aktuell wird an OpenVox Server und OpenVoxDB gearbeitet.

Ich nutze Puppet Open Source. Was muss ich tun?

Es gibt 2 Möglichkeiten:

Puppet Open Source
OpenVox

Puppet Open Source

Man muss bei Puppet die EULA akzeptieren und sich ab 25 Nodes um eine Lizenz kümmern.

OpenVox

Die Installations-Anleitung für die Migration findet man auf der
OpenVox Projekt Seite.

Da die Konfigurationspfade und -dateien, sowie die Kommandos alle identisch sind, muss keine weitere Anpassung oder Umstellung vorgenommen werden.
Auch der existierende Puppet Code funktioniert ohne Anpassungen.

Nutzer von OpenVox werden gebeten die aktuellen Pakete und Container in einer Testumgebung zu validieren.

Die weitere Entwicklung

Es gibt eine ganze Reihe von Wünschen aus der Community, die bisher leider nicht umgesetzt wurden.
Zuerst möchte man aktuelle Betriebssystemversionen unterstützen. Hierbei hatte die Community oft nach Paketen für Debian stable und testing gefragt.

Im nächsten Schritt soll der Code von alten Artefakten befreit werden.
Nicht mehr gepflegte Ruby Erweiterungen sollen durch moderne Implementierungen ersetzt werden.
Der Code Zoo (Jruby, Clojure, Java) soll aufgeräumt werden.

Wie kann man helfen?

Die Puppet Community möchte sich als Professionelles Open Source Projekt etablieren.
Dazu sucht man die finanzielle Unterstützung von Usern, Firmen und öffentlichen Einrichtungen für Open Source Arbeit.

Auch individuelle Zuarbeit ist gerne gesehen. Es muss Dokumentation geschrieben werden, die Pakete müssen getestet werden.
Die Community braucht die Unterstützung der Anwender, sei es durch Zuarbeit, Code Reviews oder Bug Reports.

Im nächsten Schritt benötigt man Spiegel Server, Bandbreite und Storage.

Professioneller Support

Open Source Support ist für viele Kunden ein wichtiger Aspekt bei der Wahl der Mittel.
Für OpenVox gibt es bereits jetzt schon 2 Firmen, die Support anbieten:

Wir von der betadots GmbH unterstützen Kunden bei Planung, Installation, Updates von OpenVox, Puppet Open Source und Puppet Enterprise und kümmern uns um die Weiterbildung der Mitarbeiter im Umfeld von Git und GitLab, Foreman/Katello, Puppet/OpenVox und Linux HA.

In Amerikanischem Raum wird die Firma overlookinfratech Ansprechpartner für OpenVox Support sein.

Weitere Firmen werden auf der Support Übersicht von OpenVox gelistet werden.

Zusammenfassung

OpenVox ist ein drop-in Ersatz für Puppet Open Source.

Die Community hat gezeigt, dass sie den Wunsch nach Weiterentwicklung hat und in der Lage ist diese Arbeiten durchzuführen.

Professioneller Support und Trainings stehen in den USA und in Europa zur Verfügung.

Open Source ist die Basis für den Erfolg vieler Unternehmen. Deshalb: Unterstützt Eure Community.

Für weitere Informationen stehen wir gerne zur Verfügung.

Datacenters need automation

betadots GmbH

The Ruby side of Puppet - Part 3 - Custom Types and Providers

Martin Alfke — Mon, 20 Jan 2025 18:50:29 +0000

This is the final post in a three-part serie, covering the concepts and best practices for extending Puppet using custom facts, custom functions and custom types and providers.

Part 1 explores how to create custom facts, which allow nodes to send information to the Puppet Server.

Part 2 discusses building custom functions to process data or execute specific tasks.

Part 3 (this post) focuses on custom types and providers, allowing you to extend Puppet's DSL and control system resources.

Types are at the core of Puppet’s declarative DSL. Types describe the desired state of the system, targeting specific, configurable parts (sometimes OS-specific). Puppet provides a set of core types, available with any Puppet agent installation.

Some of the core types include:

file
user
group
package
service

Why Create Custom Types and Providers:

There are several reasons to develop custom types and providers:

Avoid relying on the often unreliable or hard-to-maintain exec resources
Manage an application that requires CLI commands for configuration.
Handle configurations with highly specific syntax that existing Puppet types cannot manage (like file or concat).

Content:

General concepts
Types and providers in modules
Type description
Provider implementation
Using custom types in Puppet DSL
Resource API
Summary

General concepts

A custom type consists of two main parts:

Type definiton: This defines how the type will be used in the Puppet DSL.
Provider implementation(s): One or more providers per type define how to interact with the system to manage resources.

Type definition

The type describes how Puppet resources are declared in the DSL. For example, to manage an application’s configuration:

app_config { <setting>:
  ensure   => present,
  property => <value>,
  param    => <app_args>,
  provider => <auth_method>,
}

In the type definition, you typically have a namevar (a key identifier) and mutiple parameters and properties.

Namevar: The key identifying the resource instance in the type declaration.
Properties: These represent something measurable on the target system, such as a user’s UID or GID, or an application’s config value.
Parameters: Parameters influence how Puppet manages a resource but don’t directly map to something measurable on the system. For example, manage_home in the user type is a parameter that affects Puppet’s behavior but isn’t a property of the user account.

The difference between parameters and properties is nicely described on the Puppet Type/Provider development page:

Properties:

"Properties correspond to something measurable on the target system. For example, the UID and GID of a user account are properties, because their current state can be queried or changed. In practical terms, setting a value for a property causes a method to be called on the provider."

Parameters:

"Parameters change how Puppet manages a resource, but do not necessarily map directly to something measurable. For example, the user type’s managehome attribute is a parameter — its value affects what Puppet does, but the question of whether Puppet is managing a home directory isn’t an innate property of the user account."

In our example, the property is the value of the config setting, whereas the parameter specifies application attributes.

Provider implementations

Providers define the mechanisms for managing the state of the resources described by types. They handle:

Determining whether the resource already exists.
Creating or removing resources.
Modifying resources.
Optionally, listing all existing resources of the type.

Types and providers in modules

Custom types and providers are placed within the lib/puppet directory of a module.

The type file is located in lib/puppet/type and named after the resource type (app_config.rb in this case).
Provider implementations go into the lib/puppet/provider directory, inside a subdirectory named after the resource type. Each provider is named according to its provider implementation (e.g., ruby.rb, cli.rb, cert.rb, token.rb, user.rb)

For example:

Type: app_config
Providers:
- cert
- token
- user

# <modulepath>/<modulename>
modules/application/
    \- lib/
        \- puppet/
            |- type/
            |    \- app_config.rb
            \- provider/
                \- app_config/
                    |- cert.rb
                    |- token.rb
                    \- user.rb

Type description

New custom types can be crated using two different API versions.
APIv1 is the old, classic way using getters and setters within the providers.
APIv2 is a new implementation which is integrated into PDK - this implementation is also called Resource API.

In the next section we introduce the APIv1 implementation. The Resource API implementation is handled later in this document.

APIv1

The traditional method uses Puppet::Type.newtype, which defines the type.
It is recommended to add the type documentation into the code.
This allows users to run puppet describe <type> on their system to display the documentation.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  # ... the code ...
end

Management

The most important feature of a type is to add or remove something from the system.
This is usually handled with the ensure property.
To enable ensure, a single line is needed: ensurable

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
end

Namevar

When declaring the type, one must specify a title - one could refer to this as a type instance identifier.
Usually this reflects for something the type is managing.

user { 'betadots': # <- title
}

The most simple implementation is to add a parameter with the name :name.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
end

Passing the namevar: true key to the parameter is another way to identify a namevar:

newparam(:key, namevar: true) do
  desc 'The app config setting key to manage. see app_cli conf --help'
end

Properties

Next we add the other property. Each property can be validated by its content.
In our demo case we expect a string value.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value)do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
end

Besides this one can provide specific valid values which are validated automatically:

newproperty(:enable) do
  newvalue(:true)
  newvalue(:false)
end

Please note that arrays as property values are validated in a different way:

From Puppet custom types website:

By default, if a property is assigned multiple values in an array:

It is considered in sync if any of those values matches the current value.

If none of those values match, the first one is used when syncing the property.

If all array values should be matched, the property needs the
array_matchingto be set to :all. The default value is :first

newproperty(:flags, :array_matching => :all) do
  # ...
end

Accessing values can be done in two ways:

should for properties, value for parameters
value for both, properties and parameters

We prefer to use the explizit methods as this makes it more clear, whether we deal with a property or a parameter.

Parameters

Parameters are defined in a similar way:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        cli_args => ['-p'], # persistency
      }
  }
  ensurable
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value) do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI options to use during command execution."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s is not a an array" % value
      end
    end
    defaultto ['-p']
  end
end

Boolean parameters

Parameter which get a bool value should be handled in a different way to avoid code repetition

require 'puppet/parameter/boolean'
# ...
newparam(:force, :boolean => true, :parent => Puppet::Parameter::Boolean)

Automatic relationships

Within the type we can specify soft dependencies between different types.

E.g. the app_cli should use a user which must be available on the system

autorequire(:user) do
  ['app']
end

From now on the new custom type can already be used in Puppet DSL, the compiler will create a catalog but the agent will produce an error, as there are no functional providers.

Agent side prerun evaluation

It is possible to have the agent first check some things prior the catalog will be applied.
This is what the :pre_run_check method can be used for

def pre_run_check
  File.exist?('/opt/app/bin/app.exe') && raise Puppet::Error, "App not installed"
end

Features

When using multiple providers (similar to the package resource) we want to be sure that the provider has all required implementations (features).
A type can require a feature:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Manage application config. There are three ways
    to authenticate: cert, token, user. cert is the default
    provider.

    Example:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        file     => '/opt/app/etc/app.cfg',
        cli_args => ['-p'], # persistency
      }
  }
  ensurable
  # global feature
  # feature :cli, "The cli feature requires some parameters", :methods => [:cli]
  newparam(:name) do
    desc "The app config setting to manage. see app_cli conf --help"
  end
  newproperty(:value) do
    desc "The config value to set."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s is not a valid value" % value
      end
    end
  end
  # The property config_file must be set when using the cli 
  #   option to configure app.
  # The cli provider will check for a feature.
  newproperty(:file, :required_features => %w{cli}) do
    desc "The config file to use."
    validate do |value|
      unless (File.expand_path(value) == value)
        raise ArgumentError, "%s is not an absolute path" % value
      end
    end
    defaultto: '/opt/app/etc/app.cfg'
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI options to use during command execution."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s is not a an array" % value
      end
    end
    defaultto: ['-p']
  end
end

Provider implementation

Once the type is defined, the provider must handle how the resource is managed. Providers typically implement methods to:

Check if the resource exists (exists?).
Create a new resource (create).
Read existing resource properties (prefetch or a getter)
Modify a resource (flush or a setter).
Delete a resource (destroy).

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider."
end

It is also possible to re-use and extend existing providers classes.
Common code can be placed inside a generic provider (lib/puppet/provider/app_config.rb).

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli, :parent => Puppet::Provider::App_config) do
  desc "The app_config types cli provider reuses shared/common code from app_config.rb."
end

If you want to add shared functionality to multiple providers, you can place your code into the PuppetX module directory: lib/puppet_x/<company_name>/<unique class name>.

# lib/puppet/provider/app_config/cli.rb
require_relative '../../puppet_x/betadots/app_api.rb'
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider reuses shared/common code from app_config.rb."
end

A new provider can be created by inheriting from and extending an existing provider:

# lib/puppet/provider/app_config/token.rb
Puppet::Type.type(:app_config).provide(:token, :parent => :api, :source => :api) do
  desc "The app_config types token provideris an extension to the api provider."
end

Additionally one can reuse any provider from any type:

# lib/puppet/provider/app_config/file.rb
Puppet::Type.type(:app_config).provide(:file, :parent => Puppet::Type.type(:ini_setting).provider(:ruby)) do
  desc "The app_config types token provideris an extension to the api provider."
end

Selection of provider

The Puppet Agent must know which provider to use, if multiple providers are present.
You can use of the provider meta parameterr or let providers perform checks to determine if they are valid for the system. The options include:

Check	Example	Description
commands	`commands :app => "/opt/app/bin/app.exe"`	Valid if the command `/opt/app/bin/app.exe` exists. The command can be later used using `:app`.
confine - exists	`confine :exists => "/opt/app/etc/app.conf"`	Valid if the file exists.
confine - bool	`confine :true => /^10\./.match(%x{/opt/app/bin/app.exec version})`	Valid if Version is 10.x
confine - Fact	`confine 'os.family' => :debian`	Valid if the fact value matches
confing - feature	`confine :feature => :cli`	Valid if the provider has the feature
defaultfor	`defaultfor 'os.family' => :debian`	Use this provider as default for Debian-based systems.

Reading and applying configuration

The provider needs the ability to create, read, update and delete individual configuration states (CRUD API).

Each configuration property needs a getter (read) and a setter (modify) method.

# lib/puppet/provider/app_config/cli.rb
#   app_config: { 'enable_logging':
#     ensure   => present,
#     value    => true,
#   }
#
Puppet::Type.type(:app_config).provide(:cli) do
  desc "The app_config types cli provider."

  # confine to existing command
  commands :app => "/opt/app/bin/app.exe"

  def exists?
    @result = app_cli('list').split(%r{\n}).grep(%r{^#{resource[:key]}})
    if @result.length > 1
      raise ParserError, 'found multiple config items found, please fix this'
    end
    return false if @result.empty?
    return true unless @result.empty?
  end

  def create
    app(['set', resource[:name], resource[:value]])
  end

  def destroy
    app(['rm', resource[:name]])
  end

  # getter
  def value()
    @result[0].split[1]
  end

  # setter
  def value=(value)
    app(['set', resource[:name], resource[:value]])
  end
end

It is recommended to also create the instances class method, which can collect all instances of a resource type into a hash.
The namevar is the hash key, which has a hash of paramters and properties as value.

require 'yaml'
def self.instances
  instances = []
  YAML.load(app_cli('list')).each do |key, value|
    attributes_hash = { key: key, ensure: :present, value: value, name: key}
    instances << new(attributes_hash)
  end
  instances
end

Sometimes it is not possible to collect all instances, such as with the file resource. In such cases no instance method is defined.

The instance method is used indirectly be each type when calling prefetch to get the acutal configuration and returns the@property_hash instance variable.

def self.prefetch(resources)
  resources.each_key do |name|
    provider = instances.find { |instance| instance.name == name }
    resources[name].provider = provider if provider
  end
end

This allows the type getter and setter to read an manipulate the instance variable, instead of writing getter and setter methods for each type.
This behavior is added by declaring the mk_resource_methods class method.

Once this is implemented you can run the command puppet resource app_config to retrieve all existing configurations.

Refresh events

In some cases it is required to refresh a resource, for example, to restart a service, remount a mount, or rerun an exec resource. To allow a type/provider to react to a refresh event, it needs special handling.

Within the type the refreshable feature must be activated and a refresh definition is added.

The following examples are taken from puppet service type and provider. The feature describes, which provider definition should be executed upon refresh event.

# lib/puppet/type/services.rb
# ...
  feature :refreshable, "The provider can restart the service.",
    :methods => [:restart]
# ...
  def refresh
    # Only restart if we're actually running
    if (@parameters[:ensure] || newattr(:ensure)).retrieve == :running
      provider.restart
    else
      debug "Skipping restart; service is not running"
    end
  end
# ...

Using custom types in Puppet DSL

Once the custom type and provider are implemented, you can use them in your manifests as follows:

# Type declaration
app_config { 'enable_logging':
  ensure    => present,
  value     => true,
  require   => File['/opt/app/etc/app.cfg'],
  subscribe => Service|'app'|,
}
# Type reference
service { 'app':
  ensure    => running,
  subscribe => App_config['enable_logging'],
}
# Type declaration using lambda and splat operator
$config_hash.each |String $key, Hash $hash| {
  app_config { $key:
    * => $hash,
  }
}
# Virtual or exported resource
@@app_config { 'app_mount':
  ensure => present,
  value  => "${facts['networking']['fqdn']}/srv/app_mount",
  tag    => 'blog',
}
# Resource collector
App_config <<| tag == 'blog' |>>

Resource API

The modern resource API has one limitation: one can not refresh types!
Besides this it also consists of types and providers which must be placed at the same location as the existing implementation.

Resource API Type

The resource API type uses an attributes hash to list all parameters and properties. Within the type key one can use any of the built-in Puppet data types.

# lib/puppet/type/app_config.rb
require 'puppet/resource_api'

Puppet::ResourceApi.register_type(
  name: 'app_config',
  docs: <<-EOS,
      @summary The type to manage app config settings
      @example
      app_config { 'key':
        ensure => present,
        value  => 'value',
      }

      This type provides Puppet with the capabilities to manage our application config
    EOS
  features: []
  attributes: {
    ensure: {
      type:    'Enum[present, absent]',
      desc:    'Whether the setting should be added or removed',
      default: 'present',
    },
    name: {
      type:     'String',
      desc:     'The setting to manage',
      behavior: :namevar,
    },
    value: {
      type: 'Variant[String, Integer, Array]',
      desc: 'The value to set'
    }
  }
)

Respource API Provider

The most simple solution is to make use of the SimpleProvider class.
This provider needs up to 4 defines:

get
create
update
delete

# lib/puppet/provider/app_config/cli.rb
require 'puppet/resource_api/simple_provider'
require 'open3'

class Puppet::Provider::AppConfig::Cli < Puppet::ResourceApi::SimpleProvider
  $app_command = '/opt/app/bin/app.exe'

  # Get: fetching all readable config settings into a hash
  def get(context)
    context.debug('Returning data from command')

    @result = []
    stdout, stderr, status = Open3.capture3('/opt/app/bin/app.exe list')
    raise ParseError "Error running command: \n#{stderr}" unless status.success?
    # Missing error handling. This is just for demo
    @command_result = stdout.split(%r{\n})
    @command_result.each do |line|
      next if line == '---'
      key = line.split(':')[0]
      value = line.split(':')[1].strip
      hash = {
        'ensure': 'present',
        'name': key,
        'value': value,
      }
      @result += [ hash ]
    end
    @result
  end

  # Create: add a new config setting with name and should value
  def create(context, name, should)
    context.notice("Creating '#{name}' with #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end

  # Update: correct an existing setting with name and replace with should value
  def update(context, name, should)
    context.notice("Updating '#{name}' with #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end

  # Delete: remove a config setting
  def delete(context, name)
    context.notice("Deleting '#{name}'")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe rm #{name}")
    # Missing error handling. This is just for demo
    return true unless status.success?
  end
end

Summary

Types and providers are not complex. They basically describe the CRUD behavior of Puppet.
The type defines how to use the Puppet DSL, while the provider controls how to check and handle specific settings.

Please stop using exec for almost anything that is not super simple. In most cases, a simple type and provider will allow better analysis, error handling and control over what is happening. Besides this: custom types and providers execute faster compared to exec type usage - far more faster!

The example application and the working types and providers are available on GitHub:

The app: workshop demo app
The Puppet module with types and providers for the app: workshop demo module

App usage:

# Install APP:
git clone https://github.com/betadots/workshop-demo-app /opt/app
/opt/app/bin/app.exe

# Install Puppet module:
git clone https://github.com/betadots/workshop-demo-module -b ruby_workshop modules/app
# Type API V1
puppet resource app_config --modulepath modules
# Type Resource API
puppet resource app_config2 --modulepath modules

Happy puppetizing,
Martin

Die Ruby-Seite von Puppet - Teil 3 - Benutzerdefinierte Typen und Provider

Martin Alfke — Mon, 20 Jan 2025 18:48:32 +0000

Dies ist der letzte Beitrag in einer dreiteiligen Serie, die die Konzepte und Best Practices für die Erweiterung von Puppet mithilfe von benutzerdefinierten Fakten, benutzerdefinierten Funktionen und benutzerdefinierten Typen sowie Providern behandelt.

Teil 1 untersucht, wie man benutzerdefinierte Fakten erstellt, die es Knoten ermöglichen, Informationen an den Puppet-Server zu senden.

Teil 2 behandelt den Aufbau benutzerdefinierter Funktionen zur Verarbeitung von Daten oder zur Ausführung spezifischer Aufgaben.

Teil 3 (dieser Beitrag) konzentriert sich auf benutzerdefinierte Typen und Provider, mit denen Puppets DSL erweitert und Systemressourcen verwaltet werden können.

Typen stehen im Zentrum der deklarativen DSL von Puppet. Typen beschreiben den gewünschten Zustand des Systems und zielen auf spezifische, konfigurierbare Teile ab (manchmal betriebssystemspezifisch). Puppet bietet eine Reihe von Kern-Typen, die mit jeder Puppet-Agent-Installation verfügbar sind.

Einige der Kern-Typen umfassen:

file
user
group
package
service

Warum benutzerdefinierte Typen und Provider erstellen:

Es gibt mehrere Gründe, benutzerdefinierte Typen und Provider zu entwickeln:

Vermeiden von den oft unzuverlässigen oder schwer zu wartenden exec-Ressourcen.
Verwalten einer Anwendung, die CLI-Befehle zur Konfiguration benötigt.
Handhaben von Konfigurationen mit hochspezifischer Syntax, die bestehende Puppet-Typen nicht verwalten können (wie file oder concat).

Inhalt:

Allgemeine Konzepte
Typen und Provider in Modulen
Typbeschreibung
Provider-Implementierung
Verwendung benutzerdefinierter Typen in der Puppet DSL
Resourcen-API
Zusammenfassung

Allgemeine Konzepte

Ein Typ besteht aus zwei Hauptteilen:

Typdefinition: Diese definiert, wie der Typ in der Puppet DSL verwendet wird.
Provider-Implementierung(en): Ein oder mehrere Provider pro Typ definieren, wie mit dem System interagiert wird, um Ressourcen zu verwalten.

Typdefinition

Der Typ beschreibt, wie Puppet-Ressourcen in der DSL deklariert werden. Zum Beispiel, um die Konfiguration einer Anwendung zu verwalten:

app_config { <setting>:
  ensure   => present,
  property => <value>,
  param    => <app_args>,
  provider => <auth_method>,
}

Typdefinition haben einen Namevar (ein Schlüsselbezeichner) und mehrere Parameter und Properties (Eigenschaften).

Namevar: Der Schlüssel, der die Ressourceninstanz in der Typdeklaration identifiziert.
Properties (Eigenschaften): Diese repräsentieren etwas Messbares im Zielsystem, wie die UID oder GID eines Benutzers oder einen Konfigurationswert einer Anwendung.
Parameter: Parameter beeinflussen, wie Puppet eine Ressource verwaltet, spiegeln jedoch nicht direkt etwas Messbares im System wider. Zum Beispiel ist manage_home im user-Typ ein Parameter, der Puppets Verhalten beeinflusst, aber keine Eigenschaft des Benutzerkontos ist.

Der Unterschied zwischen Parametern und Eigenschaften wird auf der Puppet-Typ/Provider-Entwicklungsseite gut beschrieben:

Properties:

"Eigenschaften entsprechen etwas Messbarem im Zielsystem. Zum Beispiel sind die UID und GID eines Benutzerkontos Eigenschaften, da ihr aktueller Zustand abgefragt oder geändert werden kann. Praktisch bedeutet das, dass das Festlegen eines Wertes für eine Eigenschaft eine Methode im Provider aufruft."

Parameter:

"Parameter ändern, wie Puppet eine Ressource verwaltet, entsprechen jedoch nicht unbedingt direkt etwas Messbarem. Zum Beispiel ist das managehome-Attribut des Benutzertyps ein Parameter – sein Wert beeinflusst, was Puppet tut, aber die Frage, ob Puppet ein Home-Verzeichnis verwaltet, ist keine angeborene Eigenschaft des Benutzerkontos."

In unserem Beispiel ist die Eigenschaft der Wert des Konfigurationseinstellungen, während der Parameter die Anwendungsattribute angibt.

Provider-Implementierungen

Provider definieren die Mechanismen zur Verwaltung des Zustands der durch Typen beschriebenen Ressourcen. Sie behandeln:

Bestimmung, ob die Ressource bereits existiert
Erstellen oder Entfernen von Ressourcen.
Ändern von Ressourcen.
Optional das Auflisten aller vorhandenen Ressourcen des Typs.

Typen und Provider in Modulen

Benutzerdefinierte Typen und Provider werden im Verzeichnis lib/puppet eines Moduls platziert.

Die Typdatei befindet sich in lib/puppet/type und ist nach dem Ressourcentyp benannt (app_config.rb in diesem Fall).
Provider-Implementierungen gehen in das Verzeichnis lib/puppet/provider, in ein Unterverzeichnis, das nach dem Ressourcentyp benannt ist. Jeder Provider ist nach seiner Provider-Implementierung benannt (z.B. ruby.rb, cli.rb, cert.rb, token.rb, user.rb)

Beispiel:

Typ: app_config
Provider:
- cert
- token
- user

# <modulpfad>/<modulname>
modules/application/
    \- lib/
        \- puppet/
            |- type/
            |    \- app_config.rb
            \- provider/
                \- app_config/
                    |- cert.rb
                    |- token.rb
                    \- user.rb

Typbeschreibung

Neue benutzerdefinierte Typen können mit zwei verschiedenen API-Versionen erstellt werden. APIv1 ist die alte, klassische Methode, die Getter und Setter innerhalb der Provider verwendet. APIv2 ist eine neue Implementierung, die in PDK integriert ist – diese Implementierung wird auch als Resource-API bezeichnet.

Im nächsten Abschnitt stellen wir die APIv1-Implementierung vor. Die Implementierung der Ressourcen-API wird später in diesem Dokument behandelt.

APIv1

Die traditionelle Methode verwendet Puppet::Type.newtype, die den Typ definiert. Es wird empfohlen, die Typdokumentation in den Code einzufügen. Dies ermöglicht es den Benutzern, puppet describe <type> auf ihrem System auszuführen, um die Dokumentation anzuzeigen.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten
    zur Authentifizierung: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  # ... der Code ...
end

Verwaltung

Die wichtigste Funktion eines Typs besteht darin, etwas zum System hinzuzufügen oder zu entfernen. Dies wird normalerweise mit der ensure-Eigenschaft behandelt. Um ensure zu aktivieren, ist eine einzige Zeile erforderlich: ensurable

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten
    zur Authentifizierung: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
end

Namevar

Bei der Deklaration des Typs muss ein Titel angegeben werden – dies könnte man als Typinstanz-Identifikator bezeichnen. In der Regel spiegelt dies wider, wofür der Typ verantwortlich ist.

user { 'betadots': # <- Titel
}

Die einfachste Implementierung besteht darin, einen Parameter mit dem Namen :name hinzuzufügen.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten,
    sich zu authentifizieren: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "Das Anwendungsconfig-Element, das verwaltet werden soll. Siehe app_cli conf --help"
  end
end

Das Übergeben des Schlüssels namevar: true an den Parameter ist eine weitere Möglichkeit, einen Namevar zu identifizieren:

newparam(:key, namevar: true) do
  desc 'Der Schlüssel des Anwendungsconfig-Elements, das verwaltet werden soll. Siehe app_cli conf --help'
end

Eigenschaften

Als Nächstes fügen wir die andere Eigenschaft hinzu. Jede Eigenschaft kann durch ihren Inhalt validiert werden. In unserem Demo-Fall erwarten wir einen Stringwert.

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten,
    sich zu authentifizieren: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
      }
  }
  ensurable
  newparam(:name) do
    desc "Das Anwendungsconfig-Element, das verwaltet werden soll. Siehe app_cli conf --help"
  end
  newproperty(:value) do
    desc "Der zu setzende Konfigurationswert."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s ist kein gültiger Wert" % value
      end
    end
  end
end

Außerdem kann man spezifische gültige Werte angeben, die automatisch validiert werden:

newproperty(:enable) do
  newvalue(:true)
  newvalue(:false)
end

Bitte beachten, dass Arrays als Eigenschaftswerte auf andere Weise validiert werden:

Von der Website Puppet benutzerdefinierte Typen:

Standardmäßig wird eine Eigenschaft, die mehrere Werte in einem Array zugewiesen bekommt:

Als synchron betrachtet, wenn einer dieser Werte dem aktuellen Wert entspricht.

Wenn keiner dieser Werte übereinstimmt, wird der erste Wert beim Synchronisieren der Eigenschaft verwendet.

Wenn alle Array-Werte übereinstimmen sollen, muss die Eigenschaft array_matching auf :all gesetzt werden. Der Standardwert ist :first.

newproperty(:flags, :array_matching => :all) do
  # ...
end

Der Zugriff auf Werte kann auf zwei Arten erfolgen:

should für Eigenschaften, value für Parameter.
value für sowohl Eigenschaften als auch Parameter.

Wir bevorzugen die expliziten Methoden, da dies klarer macht, ob wir es mit einer Eigenschaft oder einem Parameter zu tun haben.

Parameter

Parameter werden auf ähnliche Weise definiert:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten,
    sich zu authentifizieren: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        cli_args => ['-p'], # Persistenz
      }
  }
  ensurable
  newparam(:name) do
    desc "Das Anwendungsconfig-Element, das verwaltet werden soll. Siehe app_cli conf --help"
  end
  newproperty(:value) do
    desc "Der zu setzende Konfigurationswert."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s ist kein gültiger Wert" % value
      end
    end
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI-Optionen, die während der Befehlsausführung verwendet werden sollen."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s ist kein Array" % value
      end
    end
    defaultto ['-p']
  end
end

Boolean-Parameter

Parameter, die einen booleschen Wert erhalten, sollten auf andere Weise behandelt werden, um Codewiederholungen zu vermeiden.

require 'puppet/parameter/boolean'
# ...
newparam(:force, :boolean => true, :parent => Puppet::Parameter::Boolean)

Automatische Abhängigkeiten

Innerhalb des Typs können wir weiche Abhängigkeiten zwischen verschiedenen Typen angeben.

Beispiel: Das app_cli sollte einen Benutzer verwenden, der im System verfügbar sein muss.

autorequire(:user) do
  ['app']
end

Von nun an kann der neue benutzerdefinierte Typ bereits in Puppet DSL verwendet werden, der Compiler wird ein Katalog erstellen, aber der Agent wird einen Fehler produzieren, da es keine funktionalen Provider gibt.

Vorabprüfung auf der Agentenseite

Es ist möglich, dass der Agent zunächst einige Dinge überprüft, bevor der Katalog angewendet wird. Dafür kann die Methode :pre_run_check verwendet werden.

def pre_run_check
  File.exist?('/opt/app/bin/app.exe') && raise Puppet::Error, "App nicht installiert"
end

Merkmale

Bei der Verwendung mehrerer Provider (ähnlich der Ressourcenpakete) wollen wir sicherstellen, dass der Provider alle erforderlichen Implementierungen (Merkmale - Features) hat. Ein Typ kann ein Merkmal erfordern:

# lib/puppet/type/app_config.rb
Puppet::Type.newtype(:app_config) do
  @doc = %q{Verwalten der Anwendungsconfig. Es gibt drei Möglichkeiten,
    sich zu authentifizieren: Zertifikat, Token, Benutzer. Zertifikat ist der Standard
    Provider.

    Beispiel:

      app_config: { 'enable_logging':
        ensure   => present,
        value    => true,
        file     => '/opt/app/etc/app.cfg',
        cli_args => ['-p'], # Persistenz
      }
  }
  ensurable
  # globales Merkmal
  # feature :cli, "Das CLI-Merkmal erfordert einige Parameter", :methods => [:cli]
  newparam(:name) do
    desc "Das Anwendungsconfig-Element, das verwaltet werden soll. Siehe app_cli conf --help"
  end
  newproperty(:value) do
    desc "Der zu setzende Konfigurationswert."
    validate do |value|
      unless value =~ /^\w+/
        raise ArgumentError, "%s ist kein gültiger Wert" % value
      end
    end
  end
  # Die Eigenschaft config_file muss gesetzt werden, wenn die CLI
  #   Option zur Konfiguration von App verwendet wird.
  # Der CLI-Provider wird nach einem Merkmal suchen.
  newproperty(:file, :required_features => %w{cli}) do
    desc "Die zu verwendende Konfigurationsdatei."
    validate do |value|
      unless (File.expand_path(value) == value)
        raise ArgumentError, "%s ist kein absoluter Pfad" % value
      end
    end
    defaultto '/opt/app/etc/app.cfg'
  end
  newparam(:cli_args, :array_matching => :all) do
    desc "CLI-Optionen, die während der Befehlsausführung verwendet werden sollen."
    validate do |value|
      unless value.class == Array
        raise ArgumentError, "%s ist kein Array" % value
      end
    end
    defaultto ['-p']
  end
end

Provider-Implementierung

Sobald der Typ definiert ist, muss der Anbieter steuern, wie die Ressource verwaltet wird. Anbieter implementieren typischerweise Methoden, um:

Überprüfen, ob die Ressource existiert (exists?).
Eine neue Ressource erstellen (create).
Vorhandene Ressourcenattribute lesen (prefetch oder ein Getter).
Eine Ressource ändern (flush oder ein Setter).
Eine Ressource löschen (destroy).

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli) do
  desc "Der CLI-Anbieter des app_config-Typs."
end

Es ist auch möglich, bestehende Anbieterklassen wiederzuverwenden und zu erweitern. Gemeinsamer Code kann in einem generischen Anbieter (lib/puppet/provider/app_config.rb) platziert werden.

# lib/puppet/provider/app_config/cli.rb
Puppet::Type.type(:app_config).provide(:cli, :parent => Puppet::Provider::App_config) do
  desc "Der CLI-Anbieter des app_config-Typs verwendet gemeinsamen Code aus app_config.rb."
end

Wenn eine gemeinsame Funktionalität für mehrere Anbieter hinzugefügt werden soll, kann man den Code im Puppet_X-Modulverzeichnis ablegen: lib/puppet_x/<unternehmens_name>/<eindeutiger Klassenname>.

# lib/puppet/provider/app_config/cli.rb
require_relative '../../puppet_x/betadots/app_api.rb'
Puppet::Type.type(:app_config).provide(:cli) do
  desc "Der CLI-Anbieter des app_config-Typs verwendet gemeinsamen Code aus app_config.rb."
end

Ein neuer Anbieter kann erstellt werden, indem er von einem bestehenden Anbieter erbt und diesen erweitert:

# lib/puppet/provider/app_config/token.rb
Puppet::Type.type(:app_config).provide(:token, :parent => :api, :source => :api) do
  desc "Der Token-Anbieter des app_config-Typs ist eine Erweiterung des API-Anbieters."
end

Zusätzlich kann man jeden Anbieter von jedem Typ wiederverwenden:

# lib/puppet/provider/app_config/file.rb
Puppet::Type.type(:app_config).provide(:file, :parent => Puppet::Type.type(:ini_setting).provider(:ruby)) do
  desc "Der Datei-Anbieter des app_config-Typs ist eine Erweiterung des API-Anbieters."
end

Auswahl des Anbieters

Der Puppet-Agent muss wissen, welcher Anbieter zu verwenden ist, wenn mehrere Anbieter vorhanden sind. Man kann den provider-Meta-Parameter verwenden oder die Anbieter Überprüfungen durchführen lassen, um festzustellen, ob sie für das System gültig sind. Die Optionen umfassen:

Überprüfung	Beispiel	Beschreibung
Befehle	`commands :app => "/opt/app/bin/app.exe"`	Gültig, wenn der Befehl `/opt/app/bin/app.exe` existiert. Der Befehl kann später mit `:app` verwendet werden.
Einschränkung - existiert	`confine :exists => "/opt/app/etc/app.conf"`	Gültig, wenn die Datei existiert.
Einschränkung - bool	`confine :true => /^10\./.match(%x{/opt/app/bin/app.exec version})`	Gültig, wenn die Version 10.x ist.
Einschränkung - Fakt	`confine 'os.family' => :debian`	Gültig, wenn der Faktwert übereinstimmt.
Einschränkung - Funktion	`confine :feature => :cli`	Gültig, wenn der Anbieter die Funktion hat.
defaultfor	`defaultfor 'os.family' => :debian`	Anbieter als Standard für Debian-basierte Systeme.

Lesen und Anwenden von Konfigurationen

Der Anbieter benötigt die Fähigkeit, einzelne Konfigurationszustände zu erstellen, zu lesen, zu aktualisieren und zu löschen (CRUD-API).

Jede Konfigurationseigenschaft benötigt eine getter- (lesen) und eine setter- (ändern) Methode.

# lib/puppet/provider/app_config/cli.rb
#   app_config: { 'enable_logging':
#     ensure   => present,
#     value    => true,
#   }
#
Puppet::Type.type(:app_config).provide(:cli) do
  desc "Der CLI-Anbieter des app_config-Typs."

  # Einschränkung auf bestehenden Befehl
  commands :app => "/opt/app/bin/app.exe"

  def exists?
    @result = app_cli('list').split(%r{\n}).grep(%r{^#{resource[:key]}})
    if @result.length > 1
      raise ParserError, 'Mehrere Konfigurationselemente gefunden, bitte beheben Sie dies'
    end
    return false if @result.empty?
    return true unless @result.empty?
  end

  def create
    app(['set', resource[:name], resource[:value]])
  end

  def destroy
    app(['rm', resource[:name]])
  end

  # getter
  def value()
    @result[0].split[1]
  end

  # setter
  def value=(value)
    app(['set', resource[:name], resource[:value]])
  end
end

Es wird empfohlen, auch die instances-Klassenmethode zu erstellen, die alle Instanzen eines Ressourcentyps in einem Hash sammeln kann. Der Namevar ist der Hash-Schlüssel, der einen Hash von Parametern und Eigenschaften als Wert hat.

require 'yaml'
def self.instances
  instances = []
  YAML.load(app_cli('list')).each do |key, value|
    attributes_hash = { key: key, ensure: :present, value: value, name: key}
    instances << new(attributes_hash)
  end
  instances
end

Manchmal ist es nicht möglich, alle Instanzen zu sammeln, z. B. beim file-Ressourcentyp. In solchen Fällen wird keine instance-Methode definiert.

Die instance-Methode wird indirekt von jedem Typ verwendet, wenn prefetch aufgerufen wird, um die aktuelle Konfiguration zu erhalten und die @property_hash-Instanzvariable zurückzugeben.

def self.prefetch(resources)
  resources.each_key do |name|
    provider = instances.find { |instance| instance.name == name }
    resources[name].provider = provider if provider
  end
end

Dies ermöglicht es dem Typ, Getter und Setter zu verwenden, um die Instanzvariable zu lesen und zu manipulieren, anstatt für jeden Typ getter- und setter-Methoden zu schreiben. Dieses Verhalten wird durch die Deklaration der mk_resource_methods-Klassenmethode hinzugefügt.

Sobald dies implementiert ist, kann man den Befehl puppet resource app_config ausführen, um alle vorhandenen Konfigurationen abzurufen.

Refresh-Ereignisse

In einigen Fällen ist es erforderlich, eine Ressource zu aktualisieren (refresh), beispielsweise um einen Dienst neu zu starten, ein Laufwerk erneut zu mounten oder eine Exec-Ressource erneut auszuführen. Damit ein Typ/Anbieter auf ein Aktualisierungsereignis reagieren kann, ist eine spezielle Behandlung erforderlich.

Innerhalb des Typs muss die refreshable-Funktion aktiviert werden, und eine refresh-Definition wird hinzugefügt.

Die folgenden Beispiele stammen vom Puppet service Typ und Anbieter. Die Funktion beschreibt, welche Anbieterdefinition bei einem Aktualisierungsereignis ausgeführt werden soll.

# lib/puppet/type/services.rb
# ...
  feature :refreshable, "Der Anbieter kann den Dienst neu starten.",
    :methods => [:restart]
# ...
  def refresh
    # Nur neu starten, wenn wir tatsächlich laufen
    if (@parameters[:ensure] || newattr(:ensure)).retrieve == :running
      provider.restart
    else
      debug "Neustart überspringen; Dienst läuft nicht"
    end
  end
# ...

Verwendung benutzerdefinierter Typen in Puppet DSL

Sobald der benutzerdefinierte Typ und Anbieter implementiert sind, kann man diesen in Puppet Manifesten wie folgt verwenden:

# Typdeklaration
app_config { 'enable_logging':
  ensure    => present,
  value     => true,
  require   => File['/opt/app/etc/app.cfg'],
  subscribe => Service['app'],
}
# Typreferenz
service { 'app':
  ensure    => running,
  subscribe => App_config['enable_logging'],
}
# Typdeklaration mit Lambda und Splat-Operator
$config_hash.each |String $key, Hash $hash| {
  app_config { $key:
    * => $hash,
  }
}
# Virtuelle oder exportierte Ressource
@@app_config { 'app_mount':
  ensure => present,
  value  => "${facts['networking']['fqdn']}/srv/app_mount",
  tag    => 'blog',
}
# Ressourcen-Sammler
App_config <<| tag == 'blog' |>>

Resourcen-API

Die moderne Ressourcen-API hat eine Einschränkung: Man kann Typen nicht aktualisieren! Außerdem besteht sie aus Typen und Anbietern, die am selben Ort wie die bestehende Implementierung platziert werden müssen.

Ressourcen-API-Typ

Der Ressourcen-API-Typ verwendet einen Attributs-Hash, um alle Parameter und Eigenschaften aufzulisten. Innerhalb des type-Schlüssels kann man jeden der integrierten Puppet-Datentypen verwenden.

# lib/puppet/type/app_config.rb
require 'puppet/resource_api'

Puppet::ResourceApi.register_type(
  name: 'app_config',
  docs: <<-EOS,
      @summary Der Typ zur Verwaltung von App-Konfigurationseinstellungen
      @example
      app_config { 'key':
        ensure => present,
        value  => 'value',
      }

      Dieser Typ bietet Puppet die Möglichkeit, unsere Anwendungsconfig zu verwalten.
    EOS
  features: []
  attributes: {
    ensure: {
      type:    'Enum[present, absent]',
      desc:    'Ob die Einstellung hinzugefügt oder entfernt werden soll',
      default: 'present',
    },
    name: {
      type:     'String',
      desc:     'Die Einstellung, die verwaltet werden soll',
      behavior: :namevar,
    },
    value: {
      type: 'Variant[String, Integer, Array]',
      desc: 'Der Wert, der gesetzt werden soll'
    }
  }
)

Ressourcen-API-Anbieter

Die einfachste Lösung ist die Verwendung der Klasse SimpleProvider. Dieser Anbieter benötigt bis zu 4 Definitionen:

get
create
update
delete

# lib/puppet/provider/app_config/cli.rb
require 'puppet/resource_api/simple_provider'
require 'open3'

class Puppet::Provider::AppConfig::Cli < Puppet::ResourceApi::SimpleProvider
  $app_command = '/opt/app/bin/app.exe'

  # Get: Alle lesbaren Konfigurationseinstellungen in einen Hash abrufen
  def get(context)
    context.debug('Gibt Daten vom Befehl zurück')

    @result = []
    stdout, stderr, status = Open3.capture3('/opt/app/bin/app.exe list')
    raise ParseError "Fehler beim Ausführen des Befehls: \n#{stderr}" unless status.success?
    # Fehlende Fehlerbehandlung. Dies dient nur zur Demo
    @command_result = stdout.split(%r{\n})
    @command_result.each do |line|
      next if line == '---'
      key = line.split(':')[0]
      value = line.split(':')[1].strip
      hash = {
        'ensure': 'present',
        'name': key,
        'value': value,
      }
      @result += [ hash ]
    end
    @result
  end

  # Create: Eine neue Konfigurationseinstellung mit Name und Sollwert hinzufügen
  def create(context, name, should)
    context.notice("Erstelle '#{name}' mit #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Fehlende Fehlerbehandlung. Dies dient nur zur Demo
    return true unless status.success?
  end

  # Update: Eine vorhandene Einstellung mit Namen korrigieren und durch den Sollwert ersetzen
  def update(context, name, should)
    context.notice("Aktualisiere '#{name}' mit #{should[:value]}")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe set #{name} #{should[:value]}")
    # Fehlende Fehlerbehandlung. Dies dient nur zur Demo
    return true unless status.success?
  end

  # Delete: Eine Konfigurationseinstellung entfernen
  def delete(context, name)
    context.notice("Lösche '#{name}'")
    _stdout, _stderr, status = Open3.capture3("/opt/app/bin/app.exe rm #{name}")
    # Fehlende Fehlerbehandlung. Dies dient nur zur Demo
    return true unless status.success?
  end
end

Zusammenfassung

Typen und Anbieter sind nicht komplex. Sie beschreiben im Grunde das CRUD-Verhalten von Puppet. Der Typ definiert, wie man die Puppet DSL verwendet, während der Anbieter steuert, wie spezifische Einstellungen überprüft und behandelt werden.

Eine exzessive Nutzung der exec Resource ist zu vermweiden. In den meisten Fällen ermöglicht ein einfacher Typ und Anbieter eine bessere Analyse, Fehlerbehandlung und Kontrolle über das, was passiert. Darüber hinaus: benutzerdefinierte Typen und Anbieter führen schneller aus im Vergleich zur Verwendung des exec-Typs - viel schneller!

Die Beispielanwendung und die funktionierenden Typen und Anbieter aus diesem Posting sind auf GitHub verfügbar:

Die App: Workshop-Demo-App
Das Puppet-Modul mit Typen und Anbietern für die App: Workshop-Demo-Modul

App-Nutzung:

# APP installieren:
git clone https://github.com/betadots/workshop-demo-app /opt/app
/opt/app/bin/app.exe

# Puppet-Modul installieren:
git clone https://github.com/betadots/workshop-demo-module -b ruby_workshop modules/app
# Type API V1
puppet resource app_config --modulepath modules
# Type Resource API
puppet resource app_config2 --modulepath modules

Viel Spaß beim Puppetisieren,

Martin

The Ruby side of Puppet - Part 2 - Custom Functions

Martin Alfke — Tue, 14 Jan 2025 16:04:08 +0000

This is the second post in a series of three, covering the concepts and best practices for extending Puppet using custom facts, custom functions and custom types and providers.

Part 1 covers Custom Facts, which explain how a node can provide information to the Puppet Server.

Part 2 (this post) focuses on Custom Functions, demonstrating how to develop functions for data processing and execution.

Part 3 covers Custom Types and Providers, showing how to extend Puppet's DSL functionality.

Custom Functions in Puppet:

Functions in Puppet are executed on the Puppet server, specifically inside the compiler. These functions have access to all facts and Puppet variables, as long as they exist within the function’s namespace.

There are two main types of functions in Puppet:

Statement functions
Return value functions

Statement functions can interact with Puppet internals. The most common statement functions are include or contain, which add classes to the catalog, or noop and notice, which set all resources into simulation mode or print entries into the Puppet server log file.

Return value functions such as lookup or read_url, return values from Hiera or external sources like web servers.
Some return value functions can also manipulate data (each, filter, map, reduce).

Why Create Custom Functions?

Here are some reasons you might develop custom functions in Puppet:

Writing a custom Hiera lookup function
Converting data structures for specific needs

Content:

Function development
Functions in Modules
General API
Dispatcher and Define
Using functions in Puppet DSL
Summary

Function development

Since functions are executed only at compile time, it is advisable to first focus on their core functionality before integrating them with the Function API. This allows you to test the logic with local data, avoiding issues on the Puppet server that could break compilation or even crash the server.

For example, imagine you have a data structure that outlines various aspects of a system's configuration. Using a function, you can expose specific parts of this data structure as variables.

Example Data Structure:

server_hash:
  'postfix':
    'config':
      'transport': 'mta'
      'ssl': true
  'apache':
    'config':
      'vhost': 'server.domain.tld'
      'document_root': '/srv/www/html/server.domain.tld'
      'ssl': true
      'proxy_pass': 'http://localhost:8080'
  'tomcat':
    'config':
      'application': 'crm'
      'service': 'frontend'

Assume that this data structure comes from Hiera, and you want to allow other teams, who may write their own modules, to reuse this data without performing additional lookups. This would allow you to modify the data structure internally while still providing teams with the required information.

Example Function:

# test data
server_hash = {
  'postfix' => {
    'config' => {
      'transport' => 'mta',
      'ssl' => true
    }
  },
  'apache' => {
    'config' => {
      'vhost' => 'server.domain.tld',
      'document_root' => '/srv/www/html/server.domain.tld',
      'ssl' => true,
      'proxy_pass' => 'http://localhost:8080'
    }
  },
  'tomcat' => {
    'config' => {
      'application' => 'crm',
      'service' => 'frontend'
    }
  }
}
# function definition
def read_service_config(data, service)
  data[service]
end
# testing the function
puts read_service_config(server_hash, 'tomcat')

Functions in Modules

Puppet provides an API for creating custom functions. To ensure the Puppet agent can locate these functions, they must be placed in specific directories within a module.

Files inside the module's lib directory are synchronized across all agents. Note that it is not possible to limit which files are synchronized.

Custom functions can be added to one of two directory structures:

lib/puppet/parser/functions - Functions API v1
lib/puppet/functions - Functions API v2

The newer Functions API v2 supports subdirectories, allowing you to namespace your functions. The filename should match the function name.

In this post, we’ll focus on the modern API v2.

Example Directory Structure:

# <modulepath>/stdlib/
  \- lib/
    \- puppet/
      \- functions/
        \- stdlib/
          \- to_yaml.rb

# stdlib/lib/puppet/functions/stdlib/to_yaml.rb
Puppet::Functions.create_function(:'stdlib::to_yaml') do
  # ...
end

We recommend prefixing custom functions with the module name to avoid naming conflicts.

General API

To create custom functions in Puppet, use the following structure:

Example:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  # ...
end

Note that the namespace must be identical to the module name.

Dispatcher and Define

Within the do ... end of the function, you need to add a dispatcher. The dispatcher validates the input data and links it to a named definition, which executes the corresponding Ruby code.

The dispatcher can check data types using param and map values to Ruby symbols. The definition uses the dispatcher’s name and the parameters provided in the dispatcher.

The final command in the definition determines the return value.

Example Dispatcher for an IPv4 Address:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end
end

Here’s a table of common parameter methods used in dispatchers:

Method	Description
`param` or `required_param`	A mandatory argument. May occur multiple times. Position: All mandatory arguments must come first.
`optional_param`	An argument that can be omitted. You can use any number of these. When there are multiple optional arguments, users can only pass latter ones if they also provide values for the prior ones. This also applies to repeated arguments. Position: Must come after any required arguments.
`repeated_params` or `optional_repeated_params`	A repeatable argument, which can receive zero or more values. A signature can only use one repeatable argument. Position: Must come after any non-repeating arguments.
`required_repeated_param`	A repeatable argument, which must receive one or more values. A signature can only use one repeatable argument. Position: Must come after any non-repeating arguments.
`block_param` or `required_block_param`	A mandatory lambda (block of Puppet code). A signature can only use one block. Position: Must come after all other arguments.
`optional_block_param`	An optional lambda. A signature can only use one block. Position: Must come after all other arguments.

Adding a Definition:

Description	Dispatcher	Definition
Name	`dispatch :ip`	`def ip`
Parameter	`param ..., :ipaddr`	`def ip(ipaddr)`

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  require 'resolv'
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end

  def ip(ipaddr)
    Resolv.getname ipaddr
  end
end

Now additional dispatchers and definitions can be added:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  require 'socket'
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end

  dispatch :dnsname do
    param 'Regexp[/.*/]', :dnsname
  end

  dispatch :local_hostname do
    param 'Regexp[//]', :local_hostname
  end

  def ip(ipaddr)
    Addrinfo.tcp(ipaddr, '80').getnameinfo[0]
  end

  def dnsname(dnsname)
    Addrinfo.ip(dnsname).getnameinfo[0]
  end

  def local_hostname
    Socket.gethostname
  end
end

Using functions in Puppet DSL

Within Puppet the functions usually gets executed during compile time.
A special case is the deferred function which gets executed on the agent.

Let's consider two use-cases:

using resolv function on the compiler
using resolv function on the agent

Use case 1: compiler execution

The function can be used anywhere in the code:

class betadots::resolver (
  Stdlib::Fqdn $local_hostname = betadots::resolve(),
) {
  $ip_from_google = betadots::resolve('www.google.com')
}

Use case 2: agent execution

The function must be declared to run in deferred mode in Puppet. Avoid using the function within the class header, as this could potentially overwrite the function call from hiera data.

class betadots::local_resolve (
) {
  $local_api_ip = Deferred('betadots::resolve', ['api.int.domain.tld'])
}

Summary

Puppet provides a stable API for creating custom functions.
Before creating a new function, check whether a solutions already exists in Stdlib- or Extlib-Module.

Remember to prefix your functions with your module name to avoid conflicts and help identify their origin.

Happy puppetizing,
Martin

Die Ruby-Seite von Puppet - Teil 2 - Benutzerdefinierte Funktionen

Martin Alfke — Tue, 14 Jan 2025 16:01:03 +0000

Dies ist der zweite Beitrag einer dreiteiligen Serie, in der die Konzepte und Best Practices zum Erweitern von Puppet mithilfe von benutzerdefinierten Facts, benutzerdefinierten Funktionen und benutzerdefinierten Typen und Providern behandelt werden.

Teil 1 behandelt Benutzerdefinierte Facts, die erklären, wie ein Knoten Informationen an den Puppet-Server übermitteln kann.

Teil 2 (dieser Beitrag) konzentriert sich auf Benutzerdefinierte Funktionen und zeigt, wie man Funktionen für Datenverarbeitung und Ausführung entwickelt.

Teil 3 stellt dar, wie man die DSL-Funktionalität von Puppet mit neuen Typen und Provider erweitert.

Benutzerdefinierte Funktionen in Puppet:

Funktionen in Puppet werden auf dem Puppet-Server, insbesondere im Compiler, ausgeführt. Diese Funktionen haben Zugriff auf alle Facts und Puppet-Variablen, solange sie mit dem Namensraum angegeben werden.

Es gibt zwei Haupttypen von Funktionen in Puppet:

Anweisungsfunktionen
Rückgabefunktionen

Anweisungsfunktionen können mit den internen Mechanismen von Puppet interagieren. Die gebräuchlichsten Anweisungsfunktionen sind include und contain, die Klassen zum Katalog hinzufügen, oder noop und notice, die alle Ressourcen in den Simulationsmodus versetzen oder Einträge in die Protokolldatei des Puppet-Servers schreiben.

Rückgabefunktionen wie lookup oder read_url geben Werte aus Hiera oder externen Quellen wie Webservern zurück. Einige Rückgabefunktionen können auch Daten manipulieren (each, filter, map, reduce).

Warum benutzerdefinierte Funktionen erstellen?

Hier sind einige Gründe, warum man benutzerdefinierte Funktionen in Puppet entwickeln könnte:

Erstellen einer benutzerdefinierten Hiera-Suchfunktion
Umwandeln von Datenstrukturen für spezifische Anforderungen

Inhalt:

Funktion-Entwicklung
Funktionen in Modulen
Allgemeine API
Dispatcher und Definition
Verwendung von Funktionen in der Puppet-DSL
Zusammenfassung

Funktion-Entwicklung

Da Funktionen nur zur Kompilierzeit ausgeführt werden, empfiehlt es sich, sich zunächst auf ihre Kernfunktionalität zu konzentrieren, bevor sie in die Funktions-API integriert werden. Dadurch kann die Logik mit lokalen Daten getestet werden, ohne dass Probleme auf dem Puppet-Server auftreten, die die Kompilierung unterbrechen oder den Server sogar zum Absturz bringen könnten.

Als Beispiel wird eine Datenstruktur definiert, die verschiedene Aspekte der Konfiguration eines Systems darstellt. Mit einer Funktion kann man bestimmte Teile dieser Datenstruktur als Variablen verfügbar machen.

Beispiel für eine Datenstruktur:

server_hash:
  'postfix':
    'config':
      'transport': 'mta'
      'ssl': true
  'apache':
    'config':
      'vhost': 'server.domain.tld'
      'document_root': '/srv/www/html/server.domain.tld'
      'ssl': true
      'proxy_pass': 'http://localhost:8080'
  'tomcat':
    'config':
      'application': 'crm'
      'service': 'frontend'

Angenommen, diese Datenstruktur stammt aus Hiera, und man möchte anderen Teams, die ihre eigenen Module schreiben, die Wiederverwendung dieser Daten ohne zusätzliche Abfragen ermöglichen. Dadurch kann man die interne Datenstruktur ändern und die Funktion anpassen, während die Teams weiterhin die benötigten Informationen zur Verfügung gestellt bekommen.

Beispiel für eine Funktion:

# Test Daten
server_hash = {
  'postfix' => {
    'config' => {
      'transport' => 'mta',
      'ssl' => true
    }
  },
  'apache' => {
    'config' => {
      'vhost' => 'server.domain.tld',
      'document_root' => '/srv/www/html/server.domain.tld',
      'ssl' => true,
      'proxy_pass' => 'http://localhost:8080'
    }
  },
  'tomcat' => {
    'config' => {
      'application' => 'crm',
      'service' => 'frontend'
    }
  }
}
# Funktion definition
def read_service_config(data, service)
  data[service]
end
# Testen der Funktion
puts read_service_config(server_hash, 'tomcat')

Funktionen in Modulen

Puppet stellt eine API zum Erstellen benutzerdefinierter Funktionen zur Verfügung. Damit der Puppet Compiler diese Funktionen finden kann, müssen sie in bestimmten Verzeichnissen innerhalb eines Moduls abgelegt werden.

Dateien im lib-Verzeichnis des Moduls werden auf alle Agenten synchronisiert. Es ist nicht möglich, zu steuern, welche Dateien synchronisiert werden.

Benutzerdefinierte Funktionen können zu einer der beiden folgenden Verzeichnisstrukturen hinzugefügt werden:

lib/puppet/parser/functions - Funktionen API v1
lib/puppet/functions - Funktionen API v2

Die neuere Funktionen-API v2 unterstützt Unterverzeichnisse, sodass man Funktionen in Namespaces organisieren können. Der Dateiname sollte mit dem Funktionsnamen übereinstimmen.

In diesem Beitrag konzentrieren wir uns auf die moderne API v2.

Beispiel für eine Verzeichnisstruktur:

# <modulepath>/stdlib/
  \- lib/
    \- puppet/
      \- functions/
        \- stdlib/
          \- to_yaml.rb

# stdlib/lib/puppet/functions/stdlib/to_yaml.rb
Puppet::Functions.create_function(:'stdlib::to_yaml') do
  # ...
end

Wir empfehlen, benutzerdefinierte Funktionen mit dem Modulnamen zu präfixen, um Namenskonflikte zu vermeiden.

Allgemeine API

Um benutzerdefinierte Funktionen in Puppet zu erstellen, verwendet man die folgende Struktur:

Beispiel:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  # ...
end

Bitte beachten, dass der Funktions Namespace identisch zum Modulnamen (betadots) sein muss.

Dispatcher und Definition

Innerhalb des do ... end Blocks der Funktion muss ein Dispatcher und eine Definition hinzugefügt werden. Der Dispatcher validiert die Eingabedaten und verknüpft sie mit einer benannten Definition, die den entsprechenden Ruby-Code ausführt.

Der Dispatcher kann Datentypen mithilfe von param prüfen und Werte Ruby-Symbolen zuordnen. Die Definition verwendet den Namen des Dispatchers und die im Dispatcher bereitgestellten Parameter.

Der letzte Befehl in der Definition bestimmt den Rückgabewert.

Beispiel für einen Dispatcher für eine IPv4-Adresse:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end
end

Hier ist eine Tabelle mit gängigen Methoden für Parameter in Dispatchern:

Methode	Beschreibung
`param` oder `required_param`	Ein obligatorisches Argument. Kann mehrfach verwendet werden. Position: Alle Pflichtargumente müssen zuerst erscheinen.
`optional_param`	Ein Argument, das ausgelassen werden kann.
`repeated_params` oder `optional_repeated_params`	Ein wiederholbares Argument, das null oder mehr Werte annehmen kann.
`required_repeated_param`	Ein wiederholbares Argument, das einen oder mehr Werte annehmen muss.
`block_param` oder `required_block_param`	Ein obligatorisches Lambda.
`optional_block_param`	Ein optionales Lambda.

Hinzufügen einer Definition:

Beschreibung	Dispatcher	Definition
Name	`dispatch :ip`	`def ip`
Parameter	`param ..., :ipaddr`	`def ip(ipaddr)`

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  require 'resolv'
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end

  def ip(ipaddr)
    Resolv.getname ipaddr
  end
end

Nun können zusätzliche Dispatcher und Definitionen hinzugefügt werden:

# <modulepath>/betadots/lib/puppet/functions/betadots/resolve.rb
Puppet::Functions.create_function(:'betadots::resolve') do
  require 'socket'
  dispatch :ip do
    param 'Regexp[/^(\d{1,3}).(\d{1,3}).(\d{1,3}).(\d{1,3})$/]', :ipaddr
  end

  dispatch :dnsname do
    param 'Regexp[/.*/]', :dnsname
  end

  dispatch :local_hostname do
    param 'Regexp[//]', :local_hostname
  end

  def ip(ipaddr)
    Addrinfo.tcp(ipaddr, '80').getnameinfo[0]
  end

  def dnsname(dnsname)
    Addrinfo.ip(dnsname).getnameinfo[0]
  end

  def local_hostname
    Socket.gethostname
  end
end

Verwendung von Funktionen in der Puppet-DSL

Innerhalb von Puppet werden Funktionen in der Regel während der Kompilierzeit ausgeführt.
Eine besondere Ausnahme bildet die deferred function, die auf dem Agenten ausgeführt wird.

Im folgenden betrachten wir die zwei Anwendungsfälle:

Verwendung der resolve-Funktion im Compiler
Verwendung der resolve-Funktion auf dem Agenten

Anwendungsfall 1: Ausführung im Compiler

Die Funktion kann an beliebiger Stelle im Code verwendet werden:

class betadots::resolver (
  Stdlib::Fqdn $local_hostname = betadots::resolve(),
) {
  $ip_from_google = betadots::resolve('www.google.com')
}

Anwendungsfall 2: Ausführung auf dem Agenten

Die Funktion muss in deferred mode deklariert werden, um in Puppet auf dem Agenten ausgeführt zu werden. Die Funktion sollte nicht als Parameter im Header der Klasse verwendet werden, da dies möglicherweise den Funktionsaufruf mit hiera-Daten überschreiben könnte.

class betadots::local_resolve (
) {
  $local_api_ip = Deferred('betadots::resolve', ['api.int.domain.tld'])
}

Zusammenfassung

Puppet bietet eine stabile API zum Erstellen benutzerdefinierter Funktionen.
Bevor eine neue Funktion erstellt wird, bitte prüfen, ob bereits eine Lösung im Stdlib- oder Extlib-Modul vorhanden ist.

Funktionen sollten mit dem Modulnamen zu präfixen, um Konflikte zu vermeiden und deren Herkunft zu kennzeichnen.

Viel Erfolg beim Puppetisieren,
Martin

Die Ruby-Seite von Puppet - Teil 1 - Benutzerdefinierte Fakten

Martin Alfke — Wed, 01 Jan 2025 17:26:54 +0000

Dies ist der erste von drei Beiträgen, die die Konzepte und Best Practices zur Erweiterung von Puppet mit Hilfe von benutzerdefinierten Fakten, benutzerdefinierten Funktionen, benutzerdefinierten Typen und Providern behandeln.

Teil 1 (dieser Beitrag) erklärt benutzerdefinierte Fakten und wie ein Puppet Agent dem Puppet-Server Informationen bereitstellen kann.

Teil 2 konzentriert sich auf benutzerdefinierte Funktione und erläutert detailliert, wie Datenverarbeitungs- oder Ausführungsfunktionen entwickelt werden können.

Teil 3 deckt benutzerdefinierte Typen und Provider ab, die die Funktionalität der Puppet-DSL erweitern.

Benutzerdefinierte Fakten:

Es gibt mehrere Gründe, benutzerdefinierte Fakten zu entwickeln:

Wann immer ein VM-Namensschema verwendet wird, empfehlen wir, das Namensschema als benutzerdefinierten Fakt bereitzustellen.
Es ist manchmal erforderlich, den Zustand bestimmter Konfigurationen zu bestimmen oder ob bestimmte Software installiert ist.
Die Rechenzentrumsabteilung benötigt möglicherweise einen Überblick über die physische Hardware, Details zu den Hardwareanbietern und End-of-Life (EOL)-Supportdaten. Darüber hinaus kann die Finanzabteilung Informationen über die Anzahl der verwendeten kommerziellen Lizenzen und die Versionen der installierten Software benötigen.

Wann immer Informationen von einem Puppet Agenten benötigt werden, bietet Puppet die Möglichkeit, benutzerdefinierte Fakten zu implementieren.

Inhalt:

Faktenerstellung
Fakten in Modulen
Allgemeine API
Einschränkungen
Hilfsfunktionen
Zugriff auf andere Fakten
Rückgabewerte
Windows-Fakten
Weitere Konzepte
Codierungsstrategien und strukturierte Daten
Zusammenfassung

Faktenerstellung

Benutzerdefinierte Fakten können lokal auf einem Arbeitsplatz oder auf dem System entwickelt werden, auf dem der Fakt benötigt wird. Bevor der ungetestete Fakt in den Puppet-Code aufgenommen wird, kann der Faktladepfad auf zwei Arten angegeben und unabhängig gestestet werden.

Die folgende Verzeichnisstruktur wird verwendet:

ls ~/new_fact/
betadots_application_version.rb

Um den Fakt auszuführen, verwenden man einen der folgenden Befehle:

FACTERLIB=~/new_fact facter betadots_application_version
facter --custom-dir ~/new_fact betadots_application_version

Fakten in Modulen

Puppet stellt eine API zur Entwicklung benutzerdefinierter Fakten bereit. Um sicherzustellen, dass der Puppet-Agent sie findet, müssen benutzerdefinierte Fakten in einem bestimmten Verzeichnis innerhalb eines Moduls abgelegt werden.

Alle Dateien im lib-Verzeichnis von Modulen werden auf alle Agenten synchronisiert. Es ist nicht möglich, einzuschränken, welche Dateien synchronisiert werden.

Benutzerdefinierte Fakten sollten im Verzeichnis lib/facter eines Moduls abgelegt werden. Obwohl man einen beliebigen Dateinamen wählen kann, wird empfohlen, den Namen des Fakts als Dateinamen zu verwenden, um eine einfache Identifikation zu gewährleisten. Der Dateiname muss mit .rb enden.

Wir empfehlen außerdem, dem Namen des benutzerdefinierten Fakts einen Firmen- oder Abteilungspräfix hinzuzufügen.

Allgemeine API

Facter bietet eine API zum Erstellen neuer benutzerdefinierter Fakten.

Beispiel:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  setcode do
    # Code hier
  end
end

Der Block setcode do ... end enthält den gesamten Ruby-Code für den Fakt. Das Ergebnis des letzten Befehls oder der letzten Variablen wird als Rückgabewert des Fakts verwendet.

Einschränkungen

Da alle benutzerdefinierten Fakten an alle Puppet-Agenten verteilt werden, ist es wichtig sicherzustellen, dass betriebssystem- oder anwendungsspezifische Fakten nur dort ausgeführt werden, wo es erforderlich ist.

Dies wird durch Einschränkungen (confining) erreicht.

Einschränkungen können auf einfache oder strukturierte Fakten oder auf anderen Ruby-Code wie File.exist? angewendet werden.

Einfache Fakt-Einschränkung:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine kernel: 'Linux'

  setcode do
    # Code hier
  end
end

Einschränkung unter Verwendung eines Ruby-Blocks und Fakten:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine 'os' do |os|
    os['family'] == 'RedHat'
  end

  setcode do
    # Code hier
  end
end

Einschränkung unter Verwendung eines Ruby-Blocks und Ruby-Code:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    # Code hier
  end
end

Hilfsfunktionen

Facter bietet mehrere Hilfsmethoden, die verwendet werden können, ohne dass die Klasse facter erforderlich ist:

Hilfsfunktion	Beschreibung
`Facter::Core::Execution::execute`	Führt ein ausführbares Programm auf dem System mit einer Timeout-Option aus, um das Aufhängen von Facter-Läufen zu verhindern.
`Facter::Core::Execution::which`	Prüft, ob ein ausführbares Programm verfügbar ist. Funktioniert auf jedem unterstützten Betriebssystem und durchsucht die Standardpfade `ENV['PATH'] + ['/sbin', '/usr/sbin']`.
`Facter.value`	Bietet Zugriff auf den Wert eines anderen Fakts. Achtung! Loops vermeiden!

Weitere Informationen findet man in der API-Dokumentation.

Bitte beachten, dass Facter::Core::Execution::exec zugunsten von Facter::Core::Execution::execute veraltet ist. Dies ist wichtig, wenn man von älteren Versionen von Facter migriert.

Die timeout-Options-Hash kann auf zwei Arten festgelegt werden:

Gültig für den gesamten Fakt:

Facter.add('<name>', {timeout: 5}) do
  ...
end

Pro Ausführungsmethode:

Facter::Core::Execution::execute('<cmd>', options = {:timeout => 5})

Zum Beispiel, wenn eine Anwendung ihre Konfiguration über /opt/app/bin/app config zurückgibt, kann man ein Timeout von 5 Sekunden festlegen:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    Facter::Core::Execution.execute('/opt/app/bin/app config', {:timeout => 5})
  end
end

Das Ergebnis des letzten Befehls oder die Angabe einer Variable wird als Faktenergebnis verwendet.

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    config_list = Facter::Core::Execution.execute('/opt/app/bin/app config', {:timeout => 5})
    config_list
  end
end

Zugriff auf andere Fakten

Es ist möglich, dass ein benutzerdefinierter Fakt den Wert eines anderen Fakts verwendet, indem er Facter.value verwendet. Dies sollte jedoch vorsichtig geschehen, um zyklische Abhängigkeiten zwischen Fakten zu vermeiden.

In unserem Beispiel hat die Anwendung unterschiedliche Konfigurationspfade für verschiedene Betriebssysteme.

RedHat - /etc/sysconfig/application
Debian - /etc/default/application

Beispiel mit Zugriff auf einen anderen Fakt:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    app_cfg_file = case Facter.value('os')['family']
                   when 'RedHat'
                     '/etc/sysconfig/application'
                   when 'Debian'
                     '/etc/default/application'
                   end

    File.exist?(app_cfg_file)
  end

  setcode do
    Facter::Core::Execution.execute('/opt/app/bin/app config')
  end
end

Das Beispiel ruft den os-Fakt ab und verwendet den family-Schlüssel, um das Betriebssystem zu vergleichen, und wendet dann eine Logik an, um zu bestimmen, welchen Anwendungskonfigurationspfad geprüft werden soll.

Rückgabewerte

Puppet erwartet, dass benutzerdefinierte Fakten einen gültigen Werttyp zurückgeben, insbesondere Strings, Ganze Zahlen oder Arrays.
Wenn der Wert eines Fakts nil oder undef ist, wird der Fakt einfach nicht definiert.

Windows-Fakten

Facter ist eine plattformübergreifende API und stellt die gleichen Funktionen auf allen Betriebssystemen bereit. Es ist jedoch wichtig zu beachten, dass Pfade und ausführbare Dateien in einer Windows-Umgebung unterschiedlich sein können.

Mit Hilfe des Facter::Core::Execution.execute werden üblicherweise Powershell Kommandos gestartet.

Das folgende Beispiel erstellt einen Windows-Fakt:

# ~/new_fact/windows_version.rb
Facter.add(:windows_version) do
  confine osfamily: 'windows'

  setcode do
    Facter::Core::Execution.execute('reg query "HKLM\Software\Microsoft\Internet Explorer" /v svcVersion', {:timeout => 5})
  end
end

Das Beispiel verwendet den reg query-Befehl, um die Version des Internet Explorers zu erhalten.

Bitte beachten, dass die Win32 API calls seit Ruby 1.9 deprecated sind.
Man kann statt dessen die Fiddle oder andere Ruby Bibliotheken nutzen.

Weitere Konzepte

Protokollierung

Zur Analyse eines Facts kann man auf die debug Methode zugreifen.

Facter.add(:betadots_application_version) do
  setcode do
    Facter.debug("Custom fact 'betadots_application_version' running")
    ...
  end
end

Es sind folgende Log Level möglich: debug, info, warn, error und fatal.

Aggregierung

Eine weitere Option ist das Zusammenstellen von Facts als Hash mit Hilfe von aggregates und chunks.
Jeder chunkbenötigt einen Namen (key) und einen Wert (value), wobei der Wert entweder ein Array oder ein Hash sein muss!

Nachdem Facter alle chunks eingelesen und validiert hat, werden diese aggregiert. Als default werden die Arrays oder Hashes gemerged.
Es besteht die Möglichkeit, eine eigene Aggregierungs Funktion zu schreiben.

Wir gehen davon aus, dass unsere Beispielanwendung es uns ermöglicht, Einstellungen auszulesen: /opt/app/bin/app config <key>.

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version, :type => :aggregate) do
  confine do
    app_cfg_file = case Facter.value('os')['family']
                   when 'RedHat'
                     '/etc/sysconfig/application'
                   when 'Debian'
                     '/etc/default/application'
                   end

    File.exist?(app_cfg_file)
  end

  chunk(:config_version) do
    {:version => Facter::Core::Execution.execute('/opt/app/bin/app config version')}
  end

  chunk(:config_cache_size) do
    {:cache_size => Facter::Core::Execution.execute('/opt/app/bin/app config cache_size')}
  end

  chunk(:config_log_level) do
    {:log_level => Facter::Core::Execution.execute('/opt/app/bin/app config log_level')}
  end
end

Ergebnis:

betadots_application_version => {
  version => '1.2.4',
  cache_size => '400M',
  log_level => 'info',
}

Gewichtung

Facter ermöglicht es, den Rückgabe Wert auf unterschiedliche Arten zu ermitteln. Die einzelnen Arten bekommen eine Gewichtung, damit Facter danach entscheiden kann, welche Art (Methode) vals Rückgabewert verwendet werden soll.

Wir gehen davon aus, dass unsere Besipielanwendung entweder über systemd oder über eine andere Art gestartet wurde (PID Datei).

Hinweis: Externe Fakten haben eine Gewichtung von 1000. Man kann externe Fakten überschreiben, indem man für den Custom Fakt den gleichen Namen verwendet und eine Gewichtung über 1000 hinterlegt.

Weitere Informationen findet man in der Fact precedence Sektion der Custom facts overview Seite.

Facter.add('application_running') do
  has_weight 100

  setcode do
    Facter::Core::Execution.execute('systemctl status app')
  end
end

Facter.add('application_running') do
  has_weight 50

  setcode do
    File.exist?('/var/run/application.pid')
  end
end

Blocking und Caching

Normalerweise werden Facts bei jedem Lauf eines Puppet Agenten erneut ermittelt. Dies kann bei manchen Facts dazu führen, dass Facter mehr Zeit benötigt, bis alle Facts geladen sind. Dies gilt insbesondere, wenn Facts eine Anwendung abfragen und diese Abfrage eine hohe Last erzeugt und lange dauert oder es gibt "teure" Facts, die gar nicht benötigt werden.
Facter ermöglicht es, einen Cache zu verwenden, so dass Facts nur einmalig und danch erst nach der Lebensdauer des Cache Objektes erneut abgefragt werden und man kann Blacklisten mit unerwünschen Facts pflegen.

Die Facter Konfiguration erfolgt in /etc/puppetlabs/facter/facter.conf auf *nix Systemen oder in C:\ProgramData\PuppetLabs\facter\etc\facter.conf auf Windows.
Bitte beachten, dass die Konfiguration auf dem Puppet Agent erfolgen muss!

# /etc/puppetlabs/facter/facter.conf
facts : {
  blocklist : [ "file system", "EC2", "processors.isa" ]
  ttls : [
    { "timezone": 30 days },
    { "my-fact-group": 30 days },
  ]
}
fact-groups : {
  my-fact-group : [ "os", "ssh.version"]
}

Innerhalb des Elementes blocklist kann man entweder einen Fact, ein Subelement eines Fact Hashes oder eine Facter Gruppe angeben.
Existierende Facter Gruppen kann man aus Facter auslesen:

Facter block groups

facter --list-block-groups
EC2
  - ec2_metadata
  - ec2_userdata
file system
  - mountpoints
  - filesystems
  - partitions
hypervisors
  - hypervisors

Facter cache groups

facter --list-cache-groups
EC2
  - ec2_metadata
  - ec2_userdata
GCE
  - gce
...

Außerdem kann man eigene Facter Gruppen innerhalb der facts-group Sektion hinterlegen.

Bitte beachten, dass das Dylan Ratcliffe's facter_cache Module seit Puppet 6 nicht mehr notwendig ist.

Mehr Informationen findet man auf der Puppet Webseite im Bereich facter.conf settings.

Codierungsstrategien und strukturierte Daten

Anzahl dr Fakten redizieren
Hashes verwenden
Generischer Code und Ruby Module

Wenn ein Fakt komplexe Daten zurückgeben muss, sollten strukturierte Daten verwendet werden, um hierarchische Informationen bereitzustellen. Weitere Informationen zur Strukturierung von Daten finden Sie in der Puppet-Dokumentation zu strukturierten Daten in benutzerdefinierten Fakten.

Eine bewährte Praxis besteht darin, sich für eine Struktur zu entscheiden, wenn ein Fakt mehr als eine einzelne Zeichenfolge zurückgeben soll. Solche Daten werden in der Regel in Hashes oder Arrays organisiert, die wahlweise direkt einen Hash liefern oder über aggregates zusammengesetzt werden:

Lösung 1: Hash in Ruby:

Facter.add(:betadots_file_check) do
  setcode do
    file_list = case Facter.value(:kernel)
                when 'windows'
                  [
                    'c:/Program Data/Application/bin/app',
                    'c:/backup/state.txt',
                  ]
                when 'Linux'
                  [
                    '/opt/app/bin/app',
                    '/etc/backup/state.txt',
                  ]
                end

    result = {}

    file_list.each |name| do
      result[name] = {
        :size => File.size(name),
        :world_writable => File.world_writable?(name)
      } if File.exist?(name)
    end

    result
  end
end

Facter Ergebnis:

{
  '/opt/app/bin/app' => {
    size => 64328,
    world_writable => null,
  },
  '/etc/backup/state.txt' => {
    size => 0,
    world_writable => 438,
  }
}

Lösung 2: Hashes mit Hilfe von aggregate:

Facter.add(:betadots_file_check, :type => :aggregate) do
  file_list = case Facter.value(:kernel)
              when 'windows'
                [
                  'c:/Program Data/Application/bin/app',
                  'c:/backup/state.txt',
                ]
              when 'Linux'
                [
                  '/opt/app/bin/app',
                  '/etc/backup/state.txt',
                ]
              end

  chunk(:file_size) do
    result = {}

    file_list.each |name| do
      result[name] = { :size => File.size(name) } if File.exist?(name)
    end

    result
  end

  chunk(:file_world_writable) do
    result = {}

    file_list.each |name| do
      result[name] = { :world_writable => File.world_writable?(name) } if File.exist?(name)
    end

    result
  end
end

Facter Ergebnis:

{
  '/opt/app/bin/app' => {
    size => 64328,
    world_writable => null,
  },
  '/etc/backup/state.txt' => {
    size => 0,
    world_writable => 438,
  }
}

Ein weiteres Facter Besipiel liest das Agent Zertifikat aus und liefert die Zertifikatserweiterungen als Fact Hash:

Facter.add(:betadots_cert_extension) do
  setcode do
    require 'openssl'
    require 'puppet'
    require 'puppet/ssl/oids'

    # set variables
    extension_hash = {}
    certdir = Puppet.settings[:certdir]
    certname = Puppet.settings[:certname]
    certificate_file = "#{certdir}/#{certname}.pem"

    # get puppet ssl oids
    oids = {}
    Puppet::SSL::Oids::PUPPET_OIDS.each do |o|
      oids[o[0]] = o[1]
    end

    # read the certificate
    cert = OpenSSL::X509::Certificate.new File.read certificate_file

    # cert extensions differs if we run via agent (numeric) or via facter (names)
    cert.extensions.each do |extension|
      case extension.oid.to_s
      when %r{^1\.3\.6\.1\.4\.1\.34380\.1\.1}
        short_name = oids[extension.oid]
        value = extension.value[2..-1]
        extension_hash[short_name] = value unless short_name == 'pp_preshared_key'
      when %r{^pp_}
        short_name = extension.oid
        value = extension.value[2..-1]
        extension_hash[short_name] = value unless short_name == 'pp_preshared_key'
      end
    end

    extension_hash
  end
end

Zusammenfassung

Dieser Beitrag hat die Grundlagen zur Erstellung von benutzerdefinierten Fakten in Puppet beschrieben. Benutzerdefinierte Fakten sind ein leistungsfähiges Werkzeug, um einem Puppet-Server oder einem Management-Tool Informationen zu liefern. Facts werden in Modulen hinterlegt und sollten bei Bedarf eingeschränkt (confine) werden. Die Rückgabewerte sollten als Hash oder Array und nur in seltenen Fällen als Bool, String oder Integer ausgegeben werden.

Ein letzter Hinweis: Nur lokale Kommandos ausführen! Wenn man Remote Systeme in Kombination mit Facter nutzen möchte, müssen diese Hochverfügbar und skaliert aufgebaut sein.

Happy puppetizing,
Martin

The Ruby side of Puppet - Part 1 - Custom Facts

Martin Alfke — Wed, 01 Jan 2025 17:26:04 +0000

This is the first in a series of three posts covering the concepts and best practices for extending Puppet using custom facts, custom functions and custom types and providers.

Part 1 (this post) explains custom facts and how a node can provide information to the Puppet Server.

Part 2 focuses on custom functions, detailing how to develop data processing or execution functions.

Part 3 covers custom types and providers, extending Puppet DSL functionality.

Custom Facts:

There are several reasons to develop custom facts:

Whenever a VM naming model is used, we recommend providing the naming schema as a custom fact.
It is sometimes necessary to determine the state of certain configurations or whether specific software is installed.
The datacenter department may need an overview of physical hardware, hardware vendor details, and end-of-life (EOL) support dates. Additionally, the finance department may require information on the number of commercial licenses in use and the versions of installed software.

Whenever information from a node is needed, Puppet offers the option to deploy custom facts.

Content:

Fact Development
Facts in Modules
General API
Confining
Helpers
Accessing Other Facts
Return Values
Windows Facts
Additional Concepts
Coding Strategies and Structured Data
Summary

Fact Development

You can develop custom facts locally on a workstation or on the system where the fact is needed. Before adding the untested fact to your Puppet code, you can specify the fact load path in two ways.

The following directory structure is used:

ls ~/new_fact/
betadots_application_version.rb

To execute the fact, use one of the following commands:

FACTERLIB=~/new_fact facter betadots_application_version
facter --custom-dir ~/new_fact betadots_application_version

Facts in Modules

Puppet provides an API to develop custom facts. To ensure the Puppet agent finds them, custom facts must be placed in a specific directory within a module.

All files in the lib directory of modules are synchronized to all agents. It’s not possible to restrict which files are synchronized.

Custom facts should be placed in the lib/facter directory of a module. Although you can choose any filename, it's recommended to use the fact name as the filename for easy identification. The filename must end with .rb.

We also recommend prefixing the custom fact name with a company or department identifier.

General API

Facter provides an API to create new custom facts.

Example:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  setcode do
    # Code here
  end
end

The setcode do ... end block contains all the Ruby code for the fact. The result of the last command or variable is used as the fact’s return value.

Confining

Since all custom facts are distributed to all Puppet agents, it’s essential to ensure that OS- or application-specific facts are only executed where necessary.

This is achieved by confining facts.

Confinement can be applied to simple or structured facts or any other Ruby code, such as File.exist?.

Simple fact confinement:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine kernel: 'Linux'

  setcode do
    # Code here
  end
end

Confinement using Ruby block and facts:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine 'os' do |os|
    os['family'] == 'RedHat'
  end

  setcode do
    # Code here
  end
end

Confinement using Ruby block and Ruby code:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    # Code here
  end
end

Helpers

Facter provides several helper methods that can be used without requiring the facter class:

Helper	Description
`Facter::Core::Execution::execute`	Runs an executable on the system with a timeout option to prevent hanging facter runs.
`Facter::Core::Execution::which`	Checks if an executable is available. Works on any supported OS, searching the default paths `ENV['PATH'] + ['/sbin', '/usr/sbin']`
`Facter.value`	Provides access to the value of any other fact. Be careful to avoid loops!

More information is available in the API documentation.

Please note that Facter::Core::Execution::exec has been deprecated in favor of Facter::Core::Execution::execute. This is important when migrating from older versions of Facter.

The timeout options hash can be set in two ways:

Valid for the whole fact:

Facter.add('<name>', {timeout: 5}) do
  ...
end

Per execute method

Facter::Core::Execution::execute('<cmd>', options = {:timeout => 5})

For example, if an application returns its configuration via /opt/app/bin/app config, you can set a timeout of 5 seconds:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    Facter::Core::Execution.execute('/opt/app/bin/app config', {:timeout => 5})
  end
end

The result of the last command or a variable with content is used as fact result.

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    File.exist?('/etc/sysconfig/application')
  end

  setcode do
    config_list = Facter::Core::Execution.execute('/opt/app/bin/app config', {:timeout => 5})
    config_list
  end
end

Accessing Other Facts

It’s possible for a custom fact to leverage the value of another fact by using Facter.value.However, this should be done cautiously to avoid introducing cyclic dependencies between facts.

In our example the application has to different config paths for different OS.

RedHat - /etc/sysconfig/application
Debian - /etc/default/application

Example with access to another fact:

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version) do
  confine do
    app_cfg_file = case Facter.value('os')['family']
                   when 'RedHat'
                     '/etc/sysconfig/application'
                   when 'Debian'
                     '/etc/default/application'
                   end

    File.exist?(app_cfg_file)
  end

  setcode do
    Facter::Core::Execution.execute('/opt/app/bin/app config')
  end
end

The example retreives the os fact and uses the family key to compare the OS, then applies logic to determine which application config path to check for.

Return values

Puppet expects custom facts to return a valid value type, specifically strings, integers, booleans, or structured data (such as arrays and hashes). If the fact cannot determine a valid value, it should return nil or undef to indicate that no fact value is available.

Windows Facts

Windows-based systems support custom facts just like Linux or other Unix-like systems.
The main difference is that many Windows-specific tasks (such as checking installed software or reading from the registry)may require platform-specific Ruby code.

Within the Facter::Core::Execution.execute usually powershell commands is used.

Here's an example of a Windows custom fact that retrieves the version of Internet Explorer:

# ~/new_fact/betadots_internet_explorer_version.rb
Facter.add(:betadots_internet_explorer_version) do
  confine kernel: 'windows'

  setcode do
    Facter::Core::Execution.execute('reg query "HKLM\Software\Microsoft\Internet Explorer" /v svcVersion')
  end
end

In this example, we use the reg querycommand to check the Internet Explorer version in the Windows registry.

Please note that the Win32 API calls are deprecated since ruby 1.9.
Please consider using Fiddle or other Ruby-based libraries for interacting with the system.

Additional Concepts

Logging

It's often useful to include logging within custom facts to help with troubleshooting and development. You can log messages using Puppet's built-in logging mechanism:

Facter.add(:betadots_application_version) do
  setcode do
    Facter.debug("Custom fact 'betadots_application_version' running")
    ...
  end
end

Logging levels include debug, info, warn, error, and fatal.

Aggregates

Another option to create structured facts is the usage of chunks.
Each chunk must have a name and returns a data structure which must be of type array or hash.
After reading all chunks, the chunks will be aggregated. The default is to merge arrays or hashes.

It is possible to write a different aggregate definition.

We assume that our app can directly read config keys: /opt/app/bin/app config <key> returns the value.

# ~/new_fact/betadots_application_version.rb
Facter.add(:betadots_application_version, :type => :aggregate) do
  confine do
    app_cfg_file = case Facter.value('os')['family']
                   when 'RedHat'
                     '/etc/sysconfig/application'
                   when 'Debian'
                     '/etc/default/application'
                   end

    File.exist?(app_cfg_file)
  end

  chunk(:config_version) do
    {:version => Facter::Core::Execution.execute('/opt/app/bin/app config version')}
  end

  chunk(:config_cache_size) do
    {:cache_size => Facter::Core::Execution.execute('/opt/app/bin/app config cache_size')}
  end

  chunk(:config_log_level) do
    {:log_level => Facter::Core::Execution.execute('/opt/app/bin/app config log_level')}
  end
end

Result:

betadots_application_version => {
  version => '1.2.4',
  cache_size => '400M',
  log_level => 'info',
}

Weight

Facter offers the option to select a value based on priority of a fact.
Let's assume that an application is either started form systemd or in any other way (so it has a pid file).

Note: external facts have a built in weight value of 1000. Overriding external facts is possible by creating a fact with the same name and specifying a weight value over 1000.

More details can be found in the Fact precedence section of the Custom facts overview page.

Facter.add('application_running') do
  has_weight 100

  setcode do
    Facter::Core::Execution.execute('systemctl status app')
  end
end

Facter.add('application_running') do
  has_weight 50

  setcode do
    File.exist?('/var/run/application.pid')
  end
end

Blocking and Caching

By default, facts are recalculated each time they are queried. In certain scenarios, this might be inefficient, especially for computationally expensive facts or facts that rarely change. Facter offers a caching mechanism that allows you to persist fact values between Puppet runs.

Besides this some facts might be of no interest, but take long time to read.
In this case Facters blocklist can be used.

Configuration takes place in /etc/puppetlabs/facter/facter.conf on *nix systems or C:\ProgramData\PuppetLabs\facter\etc\facter.conf on Windows.
Please note that this configuratoin must be done on the Puppet agents!

# /etc/puppetlabs/facter/facter.conf
facts : {
  blocklist : [ "file system", "EC2", "processors.isa" ]
  ttls : [
    { "timezone": 30 days },
    { "my-fact-group": 30 days },
  ]
}
fact-groups : {
  my-fact-group : [ "os", "ssh.version"]
}

As you can see: within the blocklist we can mention the facts itself, a sub fact, or a fact group.
Existing facter groups can be read on the command line:

Facter block groups

facter --list-block-groups
EC2
  - ec2_metadata
  - ec2_userdata
file system
  - mountpoints
  - filesystems
  - partitions
hypervisors
  - hypervisors

Facter cache groups

facter --list-cache-groups
EC2
  - ec2_metadata
  - ec2_userdata
GCE
  - gce
...

Besides this we can add our own fact groups using the facts-group setting.

Please note that in the past people were using Dylan Ratcliffe's facter_cache Module. This is no longer needed as of Puppet 6, when the facter.conf settings were introduced.

The Puppet website provides more information on facter.conf settings.

Coding Strategies and Structured Data

Limit the total number of facts
Use data hashes
Generic code and Ruby modules

In more advanced cases, facts can return structured data, such as arrays or hashes. Structured facts allow for more complex data to be passed back to the Puppet server.

It is absolutely required to limit the total number of facts for a node. Main reason is the performance of PuppetDB.

Hashes or arrays can be built directly in Ruby code or via Facter aggregates.

e.g. We need the state of several files.

Solution 1: Hash in Ruby:

Facter.add(:betadots_file_check) do
  setcode do
    file_list = case Facter.value(:kernel)
                when 'windows'
                  [
                    'c:/Program Data/Application/bin/app',
                    'c:/backup/state.txt',
                  ]
                when 'Linux'
                  [
                    '/opt/app/bin/app',
                    '/etc/backup/state.txt',
                  ]
                end

    result = {}

    file_list.each |name| do
      result[name] = { 'size' => File.size(name), 'world_writable' => File.world_writable?(name) } if File.exist?(name)
    end

    result
  end
end

Will return:

{
  /opt/app/bin/app => {
    size => 64328,
    world_writable => null,
  },
  /etc/backup/state.txt => {
    size => 0,
    world_writable => 438,
  }
}

Solution 2: Hashes using aggregate:

Facter.add(:betadots_file_check, :type => :aggregate) do
  file_list = case Facter.value(:kernel)
              when 'windows'
                [
                  'c:/Program Data/Application/bin/app',
                  'c:/backup/state.txt',
                ]
              when 'Linux'
                [
                  '/opt/app/bin/app',
                  '/etc/backup/state.txt',
                ]
              end

  chunk(:file_size) do
    result = {}

    file_list.each |name| do
      result[name] = { :size => File.size(name) } if File.exist?(name)
    end

    result
  end

  chunk(:file_world_writable) do
    result = {}

    file_list.each |name| do
      result[name] = { :world_writable => File.world_writable?(name) } if File.exist?(name)
    end

    result
  end
end

Will return:

{
  /opt/app/bin/app => {
    size => 64328,
    world_writable => null,
  },
  /etc/backup/state.txt => {
    size => 0,
    world_writable => 438,
  }
}

Another example reads the Puppet agent certificate and provides the certificate extensions as a Facter hash:

Facter.add(:betadots_cert_extension) do
  setcode do
    require 'openssl'
    require 'puppet'
    require 'puppet/ssl/oids'

    # set variables
    extension_hash = {}
    certdir = Puppet.settings[:certdir]
    certname = Puppet.settings[:certname]
    certificate_file = "#{certdir}/#{certname}.pem"

    # get puppet ssl oids
    oids = {}
    Puppet::SSL::Oids::PUPPET_OIDS.each do |o|
      oids[o[0]] = o[1]
    end

    # read the certificate
    cert = OpenSSL::X509::Certificate.new File.read certificate_file

    # cert extensions differs if we run via agent (numeric) or via facter (names)
    cert.extensions.each do |extension|
      case extension.oid.to_s
      when %r{^1\.3\.6\.1\.4\.1\.34380\.1\.1}
        short_name = oids[extension.oid]
        value = extension.value[2..-1]
        extension_hash[short_name] = value unless short_name == 'pp_preshared_key'
      when %r{^pp_}
        short_name = extension.oid
        value = extension.value[2..-1]
        extension_hash[short_name] = value unless short_name == 'pp_preshared_key'
      end
    end

    extension_hash
  end
end

Summary

Custom facts are a powerful way to extend Puppet’s built-in facts with your organization’s specific information. Whether you’re retrieving application versions, monitoring system configuration, or providing custom details for a specific environment, custom facts help ensure that Puppet has the right data to apply the correct configuration.

In this post, we’ve covered the basics of custom fact development, including:

How to create custom facts in a local environment or as part of a module.
The API for creating and confining facts.
Accessing other facts and helper methods to enhance fact functionality.
Returning structured data for complex requirements.

One more advise: Run local commands only! If connections to remote systems are required you must ensure scalability and high availability.

Happy puppetizing,
Martin

Puppet best practice

Martin Alfke — Wed, 08 May 2024 08:10:51 +0000

At betadots, during our Puppet code reviews, we often receive requests for a comprehensive summary of best practices and guidelines.
In response, we've compiled this article to delve deep into Puppet's best practices and implementations.

Table of content

Control Repo
- Test your control-repo!
Hiera
Node Classification
Library Module Usage
Profiles: Doing Them right
Module Development
- General Concepts and Structure
- Parameters and Variables
- Declaration and References
- Class and Resource Ordering
- Documentation
Summary
Examples
- environment.conf
- bin/config_script.sh
- manifests/site.pp
- Hiera Config
- Simple Hiera Data
- Module Classes
- Class Spec Tests

Control Repo

The control repo's layout is crucial. We scrutinize files such as environment.conf, Puppetfile, manifests/site.pp, and hiera.yaml.

In environment.conf, we focus on settings like config_version, modulepath, and environment_timeout.

When examining Puppetfile, we emphasize library modules, versions, and sources. While Puppet forge offers convenience, mirroring library module GIT repositories internally is advisable for control and security. Using git tags in Puppetfile for module versioning facilitates pre-upgrade reviews and eliminates Puppet server internet dependencies.

Test your Control Repo

Thorough testing validates code changes effectively. Our tests include:

Linting with puppet lint
Unit testing with rspec puppet
Acceptance testing using puppet beaker or puppet litmus
Catalog diffing with catalog diff tool and catalog diff viewer

Consider adopting PDK - Puppet Development Kit or leveraging onceover.

Ensure you're using the right version of RSpec-Puppet, distinguishing between maintained projects by Rodjek and Puppet Inc..

For your reference:

The official website for puppet-lint is now located at https://puppetlabs.github.io/puppet-lint/.
Please note that the older version is no longer maintained.
Similarly, for RSpec, we advise consulting the README file in the official GitHub repository. Be cautious not to mistake it for the project by Rodjek. While they may appear similar currently, this could change in the future.

Hiera

Hiera is pivotal, covering node classification, library module API parameters, and infrastructure differences. We stress correct Hiera implementation, aligning hierarchy layers with infrastructure complexity while balancing diversity and maintainability.
All Hiera layers should rely solely on Puppet or trusted facts.

Next we analyze the data and check for data keys and their module namespace usage.

If no module namespace is used, we can be sure that all hiera values are looked up using explizit lookup function.
This has a huge impact on analyzing and understanding code and data, as class parameters are mapped to different names within hiera.

Node Classification

In manifests/site.pp, we inspect resource defaults, node declarations, and data processing.

Setting resource defaults simplifies coding, while following the roles and profiles pattern aids in managing numerous identically configured systems.

For divergent infrastructures, Hiera-based node classification offers flexibility, shifting complexity from static code to dynamic data.

We usually recommend to set the following resource defaults:

file : disable backup
exec : path parameter default
package : default provider (only needed in an environment with Windows nodes)

Another topic we look for is data processing based on node data like fqdn or network information. Here we ask to please move this to a custom fact.
Nice side effect: this will drastically reduce the compilation time, as the node calculates its own data.

Library Module Usage

Leverage upstream library modules whenever possible to solve problems efficiently. However, consider community contributions or feature requests for missing functionalities rather than internal forks or rewrites.

Modern libraries adopting the data-in-modules pattern facilitate configuration solely through Hiera data, minimizing Puppet code development.

Profiles: Doing Them Right

Profiles encapsulate infrastructure components like user management, authentication, and application deployment. Combining resource and library module declarations within profiles streamlines configuration management.

Clear profile class naming aids in identifying responsibilities and maintaining a single source of truth.

The main win over just using library module data is the capability to identify which profile declares the library class.

Any subprofile which is split into multiple files - maybe due to better readability or maintainability - must be specified being a private subprofile.
This can be achieved by placing assert_private() into the beginning of the class body.

The concept of profile module development is the same as for library modules.

Module Development

Module development involves several considerations:

General Concepts and Structure: Utilize PDK for module creation and consider modulesync for managing multiple modules.
Parameters and Variables: Emphasize class parameters for flexibility and avoid unnecessary variable mapping.
Declaration and References: Ensure resource references are in the same class as the declaration and use Puppet-lint plugins for reference checks.
Class and Resource Ordering: Maintain proper class and resource ordering for predictable execution.
Documentation: Leverage Puppet Strings for comprehensive module documentation, including parameter usage, limitations, and code examples.

General Concepts and Structure

To simply get started with a new module we recommend using PDK. It will help to ramp up the module skeleton.
If you create a class with PDK, it will also create a simple test class for it.

In case that many modules must be managed, we recommend to switch to modulesync from Voxpupuli Puppet community.

Every module must have a dedicated API.

That means that there are specific classes to be used by others, which have parameter to allow adoptions.
Any subclass which is not an API endpoint, must be marked as being private.

Testing should be done on all classes, the private ones should verify for compiler error, whereas your API classes spec tests should also cover different parameters and operating systems.

Differences between operating systems must be done via hiera data in library or profile modules.

If you find self written profile class code repeated or copied within multiple modules you must check if building an additional internal library module can be used to reduce the multiple development overhead.

Testing must at least cover linting and unit tests.
Acceptance tests need VM automation or container technologies and should be added as soon as possible.

Parameters and Variables

Class Parameters allow flexible usage of profiles. Any parameter can be validated using a Puppet Data Type. You also can create custom data types if the basic ones are not sufficient for you use case. In the past validation was often done with assert_* or validate_* functions in the class body. With data types you can specify the type of data you are expecting right in the class header and don't need to care about validation later on.

With class parameters you can specify different settings for specific use cases like access to a dev and a prod machine via hiera.

Please stop doing variable mapping!

We often see that hash parameters are not used directly but that certain elements of a hash are placed into a new variable.
This only makes sense if you need the specific value multiple times.
Rule of thumb, if you need a special piece of data more then twice, you might do a mapping.
But in most cases we recommend to not do it!

Variable mapping produces code which is hard to understand, because you must always keep in mind the original variable and the mapped variables.

Declaration and References

Using resource references, you can set default values and ordering.

We sometimes see references on resources which are not local to the class.
The class config declares the file and a class service declares the service.

Whenever someone refactors on of the classes, you must also take care on the references.

We highly recommend to make use of the reference on declaration outside of class puppet-lint plugin.

Class and Resource Ordering

We mentioned that references should be local to a declaration.
But how to ensure resource ordering?

Normally resources inside a class are always processed in the same order as they appear in the class.
There are some resource types which have soft autorequire dependecies, like user and their primary group.

In general you must ensure that classes are always done in right order.
Otherwise puppet tries to make an educated guess in which order resources and classes might be.

Classes and resources ordering should be separate from declaration.

Documentation

Classes can have many parameters. So where do you look for information on how to make use of them?
This is where Puppet strings will be used.
Puppet strings allows you to render the file REFERENCE.md using a rake task.

Additionally it offers the option to run a documentation web server with access to your completely deployed control-repository.
It will check for any file updates and update the documentation automatically.
We recommend to put a web server with user authentication in front of the puppet strings web service.

Next to class responsibility, limitations, usage and parameter documentation we recommend to document any complex code.
We sometimes see multiple usages of filter, map or reduce functions which parse and restructure data.
Any new person working on the code must be able to understand why and what is done. Data examples can be useful.

Summary

Structured and documented code simplifies Puppet management. Adopting automated testing and coding principles from reputable library modules - like the ones from Puppet Inc and Voxpupuli - enhances code quality and reliability.

Examples

Below are examples illustrating various Puppet practices.

environment.conf

config_version = 'bin/config_script.sh $environmentpath $environment'
modulepath = site:modules:$basemodulepath

bin/config_script.sh

#!/bin/bash
if [ -x /usr/bin/git ]; then
  ENVGITDIR="$1/environments/$2/.git"
  /usr/bin/git --git-dir "${ENVGITDIR}" log --pretty=format:"%h - %an, %ad : %s" -1
else
  echo "no git - environment $1"
fi
exit 0

manifests/site.pp

File {
  backup => false,
}
Exec {
  path => $facts['path'],
}
if $facts['os']['family'] == 'windows' {
  Package {
    provider => 'choco',
  }
}

# Node classification - sorted Hash
# e.g.
# classes_hash:
#   '01_base': 'profile::base'
#   '02_security': 'profile::security'
#   '12_application': 'profile::application::billing'
#   '13_service':
#     - 'profile::application::billing::backend'
#     - 'profile::application::billing::admin_ui'
#
# overwriting identifier classes:
# classes_hash:
#   '13_service': 'profile::application::billing2'
#
# disabling identifier classes:
# classes_hash:
#   '13_service': ''
#
# $element[0]: the hash key identifier
# $element[1]: the class name to load
#
$classes_hash = lookup('classes_hash', { 'value_type' => Hash, 'default_value' => {} })
$classes_hash.keys.sort.each |$key| {
  if $classes_hash[$key] != '' {
    contain $classes_hash[$key]
  } else {
    echo { $key:
      message  => "Class for ${key} on ${facts['networking']['fqdn']} is disabled",
      withpath => false,
    }
  }
}
node default {}

Hiera Config

---
version: 5

defaults:
  datadir: data
  lookup_key: eyaml_lookup_key
  options:
    pkcs7_private_key: "/etc/puppetlabs/puppet/keys/private_key.pkcs7.pem"
    pkcs7_public_key: "/etc/puppetlabs/puppet/keys/public_key.pkcs7.pem"

hierarchy:
  - name: "node yaml hierarchy"
    paths:
      - "nodes/%{trusted.certname}.yaml"
  - name: "role yaml hierarchy"
    paths:
      - "role/%{trusted.extensions.pp_role}-%{trusted.extensions.pp_env}.yaml"
      - "role/%{trusted.extensions.pp_role}.yaml"
  - name: "os yaml hierarchy"
    paths:
      - "os/%{facts.os.family}-%{facts.os.release.major}.yaml"
      - "os/%{facts.os.family}.yaml"
  - name: "zone yaml hierarchy"
    paths:
      - "zone/%{trusted.extentions.pp_zone}.yaml"
  - name: "Common"
    path: "common.yaml"

Simple Hiera Data

# data/common.yaml
---
# Node classification using classes_hash
classes_hash:
  '00_base_ssh': 'ssh'
  '01_base_nft': 'profile::nftables'
  '02_base_nft_rules': 'profile::nftables::rules::base'

# Commmon SSH data
# using saz-ssh: https://forge.puppet.com/modules/saz/ssh/readme
ssh::server_options:
    Protocol: '2'
    ListenAddress:
        - '127.0.0.0'
        - "%{::facts.networking.hostname}"
    PasswordAuthentication: 'yes'
    SyslogFacility: 'AUTHPRIV'
    UsePAM: 'yes'
    X11Forwarding: 'yes'

ssh::server::match_block:
  filetransfer:
    type: 'group'
    options:
      ChrootDirectory: '/home/sftp'
      ForceCommand: 'internal-sftp'

# Adding core resources
# requires puppetlabs-stdlib : https://forge.puppet.com/modules/puppetlabs/stdlib/readme
stdlib::manage::create_resources:
  'package':
    'nano':
      ensure: 'absent'
    'vim':
      ensure: 'present'
  'user':
    'monitoring':
      ensure: 'present'
      uid: '11225'
      gid: '11225'
      password: '!'

Module Classes

# @summary Installs and configures the application
# @author betadots GmbH
# @param parameter1 Parameter to set thing 1
# @param parameter2 Parameter to set thing 2
# @example
#     include application
#   or
#     class { 'application':
#       parameter1 => 'value,
#     }
#
class application (
  Datatype $parameter1,
  Datatype $parameter2 = 'default',
) {
  contain application::install
  contain application::config
  contain application::service

  Class['application::install']
  -> Class['application::config']
  ~> Class['application::service']
}

class application::install {
  assert_private()
  # Puppet DSL
}

...

Class Spec Tests

# spec/classes/application_spec.rb
describe 'application' do
  on_supported_os.each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts }

      it { is_expected.to compile.with_all_deps }
      it { is_expected.to contain_class('application::install')}
    end
  end
end

# spec/classes/application_install_spec.rb
describe 'application::install' do
  on_supported_os.each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts }

      it { is_expected.to compile.and_raise_error(%{private}) }
    end
  end
end

Happy puppetizing,

Martin

Scaling Puppet Infrastructure

Martin Alfke — Mon, 05 Feb 2024 09:55:29 +0000

In large environments with many nodes one must take care of Puppet server scaling.
When and if scaling is needed, depends on the number of nodes and on code complexity and size.
This article describes the different ways of tuning Puppet server infrastructure.

Table of content:

Puppet Server tuning
1. Performance tuning a single node
2. Scaling horizontally
PuppetDB tuning
Analyzing performance win
Distributed Puppet agent runs
Summary

Scaling the hard way:

Scaling beyond JRuby limits
Multiple Puppet Server instances

Puppet Server tuning

Performance tuning single node

We sometimes see the usage of horizontal scaling, even when there is no need to do it.
As scaling horizontally needs further infrastructure components like Load Balancer and Puppet Server Compilers, we usually first ask to do performance tuning on the single node instance.

A properly configured and optimized single Puppet server should be able to handle up to 2500 nodes - using default runinterval of 30 minutes.

There are three different possibilities for tuning, which sometimes rely upon each other:

Java Version

Puppet server package has a dependency on java openjdk (headless).
On most Linux distributions this will install Java 1.8.

Please ensure to upgrade to Java 17 and set the java alternative accordingly:

e.g (on Almalinux 8)

alternatives --set java /usr/lib/jvm/java-17-openjdk-17.0.9.0.9-2.el8.x86_64/bin/java

Number of JRuby instances

With its default configuration a Puppet Server spins up 2 JRuby instances.
Each instance is able to handle a single Puppet Agent request.
When there are more requests, these get queued.

Each JRuby instance needs 512 MB of Java Heap RAM.
Just to be sure: it is not possible to run more than 32 JRuby instances on a single node!

Adopting the number of JRuby instances takes place in /etc/puppetlabs/puppetserver/conf.d/puppetserver.conf within the jruby-puppet section by setting the max-active-instances parameter:

jruby-puppet: {
    ...
    max-active-instances: 8   # <------- JRuby instances
    ...
}

Please note that increasing the number of JRuby instances causes the Java process to need more Java HEAP RAM.

Puppet Server Java Heap Size

The Java engine should have enough RAM to be able to spin up and maintain the JRuby instances.
If you forget to increase Java Heap size, you will see Java out-of-memory error messages in the Puppet server logfile.

Increasing Java Heapsize takes place in /etc/default/puppetserver (Debian based) or /etc/sysconfig/puppetserver (RedHat based).

The Java Heap size is provided as a Java argument:

# /etc/[sysconfig|default]/puppetserver
JAVA_ARGS="-Xms18g -Xmx18g -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger

In this example we have set the Java Heap size (upper and lower limit) to 18 GB RAM.

# Lower limit
-Xms18g
# Upper limit
-Xmx18g

It is considered best practice to set upper and lower limit to the same value, so Java reserves the RAM upon start up.

Please note that the maximum possible value is limited by the amount of the system RAM.
If you increase the Java heap size beyond system RAM, you will find Kernel out-of-memory errors in the journal.

Besides this one must be aware that Java switches memory handling when using more than 32 GB of heap, which reduce the amount of objects. Also see the blog post from Codecentric or Atlassian.

Reserved Code Cache

When Puppet servers receive a request from a node the Puppet Code is loaded into code cache in memory.
The default value is set to 512 MB RAM.

A larger Puppet code base or complex code might need a larger Code cache setting.
Code cache is configured as Java argument in /etc/[sysconfig|default]/puppetserver:

# /etc/[sysconfig|default]/puppetserver
JAVA_ARGS="-Xms18g -Xmx18g -XX:ReservedCodeCacheSize=1024m -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger

In this example we have set the code cache size to 1024 MB RAM.

-XX:ReservedCodeCacheSize=1024m

Please note that the maximum possible value is at 2048m.

Please ensure that the Puppet server process was restarted after setting all the required configuration options.

A short mathematical sizing rule of thumb:

Java Heap size (M) = ( No of JRuby instances * 512M ) + 512M
Reserved Code Cache Size (M) = No of JRuby instances * 128M

Scaling horizontally

If a single Puppet server is not able to handle the amount of requests - e.g. in large infrastructures exceeding 2000 nodes - one has the option to set up additional compilers.

Please note that each Puppet server infrastructure component should receive the performance tuning settings!

Infrastructure setup

A high performance Puppet infrastructure consists of the following components:

CA Server
Compiler(s)
Load balancer

A Puppet agent sends the request to the Load balancer.
The load balancer passes the request to a free compiler.

From a Puppet agent point of view, the request is sent to the Load balancer and the response is received from the compiler.
This has a special meaning when it comes to SSL certificates and strict SSL validation.

We will discuss this when we come to the compiler setup.

Puppet CA Server

The Puppet CA server is a standalone single instance, which is used to spin up and maintain additional Puppet compilers.
From CA server point of view a compiler is just a node.

All new CSR's must be signed on the CA server.
When you want to scale your Puppet infrastructure the CA Server will be your existing Puppet server.

When we want to sign the compilers certificates including dns_alt_names, we must configure the CA instance, to be able to do this by modifying the /etc/puppetlabs/puppetserver/conf.d/ca.conf file:

We must allow subject alt names setting:

# /etc/puppetlabs/puppetserver/conf.d/ca.conf
certificate-authority: {
    # allow CA to sign certificate requests that have subject alternative names.
    allow-subject-alt-names: true  # <----- enable SAN cert signing

    # allow CA to sign certificate requests that have authorization extensions.
    # allow-authorization-extensions: false

    # enable the separate CRL for Puppet infrastructure nodes
    # enable-infra-crl: false
}

Please ensure that the Puppet server process was restarted after doing all the required changes.

Adding compilers

Compilers should not act as CA Server!
The CA functionality is managed in /etc/puppetlabs/puppetserver/services.d/ca.cfg

Now we need to change the settings so that a compiler does not act as CA server, but pass all CA related requests to the CA server:

# To enable the CA service, leave the following line uncommented
# puppetlabs.services.ca.certificate-authority-service/certificate-authority-service
# To disable the CA service, comment out the above line and uncomment the line below
puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service
puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service

Please not that all compilers (and the CA server) should receive the Puppet code. In most environments we see that the compilers and the CA server have a NFS mount on which the code is deployed on the CA server and used on all compilers.

The NFS share is beyond scope of this document.

The Puppet agent will not connect to a compiler but to the load balancer.

If we keep the default setup, the Puppet agent will refuse to connect to the load-balancer, if DNS ALT Names are missing in the compilers certificate.

The compiler MUST have the Load balancer DNS name configured, prior generating the CSR.

This can be achieved by adding the dns_alt_names configuration setting into 7etc/puppetlabs/puppet/puppet.conf into the agent section.

[agent]
dns_alt_names = loadbalancer.domain.tld,compiler-fqdn.domain.tld

Adding load balancer

Each request to a Puppet Server starts a compilation process with different runtimes. One should not configure the roundrobin distribution algorithm on the load balancer.

Instead we want to distribute the connections to the Puppet compiler with the least work to do - which means the one with the least connections. In HAproxy this setting is called leastconn.

Besides this you do not want to rely on Layer 3 IP connection, but on Layer 7 functionality. Within HAproxy one should add the SSL directive to each backend.

PuppetDB tuning

In most environments PuppetDB is used to store facts, reports, catalogs and exported resources.
The more nodes you have in your infrastructure, the higher the load and the memory requirement on the PuppetDB process.

Tuning PuppetDB

PuppetDB is a HTTP(S) Rest API in front of a PostgreSQL database.
One can say that PuppetDB also is a kind of web service.

Java Heap size

The most important setting is Java heap size.
Per default a PuppetDB is configured to use 512MB Heap size.

Configuration takes place in /etc/sysconfig/puppetdb (RedHat/SLES) or 7etc/default/puppetdb (Debian).

The Java Heap size is provided as a Java argument:

# /etc/[sysconfig|default]/puppetdb
JAVA_ARGS="-Xms1g -Xmx1g ...

In this example we have set the Java Heap size (upper and lower limit) to 1 GB RAM.

# Lower limit
-Xms1g
# Upper limit
-Xmx1g

It is considered best practice to set upper and lower limit to the same value, so Java reserves the RAM upon start up.

We usually receive good results with 1GB or 2GB heap site (depending on the number of nodes)

Database connection pool

Within the PuppetDB configuration file, which is located at /etc/puppetlabs/puppetdb/conf.d/database.conf, one can set the maximum numbers of idle and active PostgreSQL connections within the database section:

maximum-pool-size = 25 # default

Please note that PuppetDB uses two connection pools:

read pool
write pool

The read pool can be set individually by setting maximum-pool-size in the read-database section. The default value is 10.

The maximum-pool-size should be set to the sum of both.

Please check your PostgreSQL configuration (especially the max connection setting) prior increasing the connection pool.

Block facts

Another possibility is to lower the number of facts stored.

This can be achieved by setting facts-blocklist.

facts-blocklist = ["fact1", "fact2", "fact3"]

Command processing threads

Besides this PuppetDB lets you configure the number of threads by setting threads within the command-processing section in /etc/puppetlabs/puppetdb/conf.d/config.ini.
As default PuppetDB will use half the number of cores of the system.

Web server threads

Last but least one can set the max-threads in /etc/puppetlabs/puppetdb/conf.d/jetty.ini.
This specifies the number if parallel possible HTTP and HTTPS connections.

Scaling PuppetDB

In former times, it was best practice to have a single PuppetDB instance on the Puppet Server.
If you have the need for scaling Puppet Servers, one should consider having PuppetDB on each Puppet infrastructure node, which connect to a centralized PostgreSQL database.

Analyzing performance win

Every time one does performance tuning, one also wants to get proof that the new setting has improved performance.
The most simple check is by getting the compile times from puppetserver logfile.

awk '/Compiled catalog/ {print $12" "$14}' /var/log/puppetlabs/puppetserver/puppetserver.log

production 0.07
production 0.05
production 0.05
production 0.04
production 0.04
production 0.03
production 0.03

You can also get highest and lowest compile time

awk '/Compiled catalog/ {print $12" "$14}' /var/log/puppetlabs/puppetserver/puppetserver.log | sort | awk 'NR==1; END{print}'

production 0.03
production 0.07

If you need an average compile time the following command will be helpful

awk '/Compiled catalog/ {print $12" "$14}' /var/log/puppetlabs/puppetserver/puppetserver.log | sort | awk '{total += $2; count++ } END { print total/count }'

0.0442857

Another option is provided by the Puppet Operational Dashboards.

This will setup a Grafana frontend which reads data from an InfluxDB, which gets its content from a Telegraf agent which connects via HTTPS to the Puppet server metrics endpoints.

In Puppet Enterprise the PostgreSQL database also uses SSL certificates, which also allows fetching PostgreSQL metrics.
When running Puppet Open Source, one needs to add an additional telegraf configuration to query the PostgreSQL database using a user and password and push the data into InfluxDB.

Distributed Puppet agent runs

Within the introduction we mentioned that a single server can handle up to 2500 nodes.
This number of nodes requires that Puppet agent runs must be evenly distributed over time.

The default way for a Puppet agent is running as a daemon, which will schedule a Puppet agent run upon start and triggers the next runs every 30 minutes (runinterval config option).

Due to the reason that systems might be starting in parallel one will still see overloaded Puppet server infrastructure.
As a result one will recognize that the Puppet agent run takes very long time. If the Agent did not receive a reply from Puppet server within 5 minutes (http_read_timeout config option), it will stop the actual request and repeat again in 30 minutes.

The config options can be modified in puppet.conf config file in section agent.

Please reconsider if Puppet should run every 30 minutes.
In some infrastructure Puppet is running in noop mode as changes may only be deployed during maintenance window.
In this case one still wants the node to regular check its configuration, but maybe 4 times a day only, so people can identify within the reporting that the systems are still in desired state.
Doubling the runinterval option reduces the load on the Puppet server to its half, so the Puppet server can handle double number of nodes.

But one still will see huge and many load spikes on the Puppet server.
It still can happen that to many nodes try to request for their configuration in parallel.

We usually recommend to customers to run the agent via cron.
One run will be done at reboot and the other runs will take place as recurring cron entries.
To distribute the agent runs over time one can make use of the fqdnrand function inside Puppet. This function generates a unique number for a hostame or FQDN.

One can use the reidmv puppet_run_scheduler module to configure Puppet accordingly.

Please note that within Puppet Enterprise environments the Puppet agent on the Puppet servers must be running as agent daemon service!

Summary

Tuning and scaling a Puppet server is very simple, as it is a single Java process and a web service only.

Tuning is mostly related to the Java process and memory handling and is usually limited by the amount of RAM and numbers of CPUs being available.

Scaling can be achieved by using simple, already established solutions to distribute web traffic amongst several servers.

Distributing the Puppet agent runs ensures evenly used Puppet servers which allows more nodes to get managed.

Scaling beyond JRuby limits

We already mentioned that you can not spin up more than 32 JRuby instances.

We tried it. Be sure. It is not working at all.

But what to do if you have high end (e.g. HPC) hardware?

72 cores
248 GB RAM

Usually you want to slice the big hardware e.g using KVM or Docker and run multiple VMs or Puppet server containers for example by using Voxpupuli CRAFTY.

Normally we see such a hardware only within virtualization or container based environments or within a HPC platform.

In our specific use case the usage of KVM, Docker, Podman or Kubernetes was not possible. Don't ask.

We talked about this in the Puppet community Slack channel and people told us that the only option is to spin up multiple Puppet server instances by "some specific symlinking and copying of config files".

Just to be sure: this is not something you can configure out-of-the-box, but needs lots of changes also to Puppet internal scripts.

Multiple Puppet Server instances

You want it the hard way. Here we go..

Please note that this is just first findings, not automated and of course unsupported!

Requirements:

big hardware
multiple IP addresses on the node
- one for the main Puppet CA server
- one for the HAproxy loadbalancer
- and one for each compiler
DNS configured for each IP
Puppet CA server configured and running

Assumptions:

the main installation is used for the Puppet CA server
compilers will use copy of this installation
main CA server FQDN: puppetserver.domain.tld
compiler FQDN: puppetcompiler1.domain.tld
load balancer FQDN: puppetbalancer.domain.tld

First we need to stop the running Puppet Server, then we can copy several directories and make modifications to configuration files and scripts.
Afterwards we can start the Puppet server processes and then we can install and configure HAproxy.

systemctl stop puppetserver

Files and directories needed

Each Puppet server instance needs several files and directories copied from the main installation:

# Service config
cp -pr /etc/sysconfig/puppetserver /etc/sysconfig/puppetservercompiler1
# Puppet agent config
cp -pr /etc/puppetlabs/puppet /etc/puppetlabs/puppetcompiler1
# Puppet server config
cp -pr /etc/puppetlabs/puppetserver /etc/puppetlabs/puppetservercompiler1
# Puppet server application
cp -pr /opt/puppetlabs/server /opt/puppetlabs/servercompiler1
# Puppet server logging
cp -pr /var/log/puppetlabs/puppetserver /var/log/puppetlabs/puppetservercompiler1
# Puppet server service
cp -pr /usr/lib/systemd/system/puppetserver.service /usr/lib/systemd/system/puppetservercompiler1.service

Config files

/etc/puppetlabs/puppet/puppet.conf

The main Puppet configuration from the main Puppet CA Server should only receive an entry for the server:

puppet config set --section main server puppetserver.domain.tld

/etc/sysconfig/puppetservercompiler1

Change the paths:

...
INSTALL_DIR="/opt/puppetlabs/servercompiler1/apps/puppetserver"
CONFIG="/etc/puppetlabs/puppetservercompiler1/conf.d"
# Bootstrap path
BOOTSTRAP_CONFIG="/etc/puppetlabs/puppetservercompiler1/services.d/,/opt/puppetlabs/servercompiler1/apps/puppetserver/config/services.d/"
...

/etc/puppetlabs/puppetservercompiler1/services.d/ca.cfg

Disable CA

# To enable the CA service, leave the following line uncommented
#puppetlabs.services.ca.certificate-authority-service/certificate-authority-service
# To disable the CA service, comment out the above line and uncomment the line below
puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service
puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service

/etc/puppetlabs/puppetservercompiler1/conf.d/global.conf

Adopt path

global: {
    # Path to logback logging configuration file; for more
    # info, see http://logback.qos.ch/manual/configuration.html
    logging-config: /etc/puppetlabs/puppetservercompiler1/logback.xml
}

/etc/puppetlabs/puppetservercompiler1/conf.d/metrics.conf

Adopt the server id

metrics: {
    # a server id that will be used as part of the namespace for metrics produced
    # by this server
    server-id: puppetservercompiler1
    registries: {
        ...

/etc/puppetlabs/puppetservercompiler1/conf.d/puppetserver.conf

Adopt paths

jruby-puppet: {
    ruby-load-path: [/opt/puppetlabs/puppet/lib/ruby/vendor_ruby]

    # Adopt path
    gem-home: /opt/puppetlabs/servercompiler1/data/puppetserver/jruby-gems

    # Adopt paths
    gem-path: [${jruby-puppet.gem-home}, "/opt/puppetlabs/servercompiler1/data/puppetserver/vendored-jruby-gems", "/opt/puppetlabs/puppet/lib/ruby/vendor_gems"]

    # Adopt path
    server-conf-dir: /etc/puppetlabs/puppetcompiler1

    server-code-dir: /etc/puppetlabs/code

    # Adopt path
    server-var-dir: /opt/puppetlabs/servercompiler1/data/puppetserver

    # Adopt path
    server-run-dir: /var/run/puppetlabs/puppetservercompiler1

    # Adopt path
    server-log-dir: /var/log/puppetlabs/puppetservercompiler1

    ...
}
...

/etc/puppetlabs/puppetservercompiler1/conf.d/webserver.conf

Adopt Path and IP and/or Port

webserver: {
    access-log-config: /etc/puppetlabs/puppetservercompiler1/request-logging.xml
    client-auth: want
    ssl-host: 10.110.10.102 # <---- add compiler1 IP and/or
    ssl-port: 8141          # <---- use another port
}

/etc/puppetlabs/puppetservercompiler1/logback.xml

Adopt paths

<configuration scan="true" scanPeriod="60 seconds">
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%t] [%c{2}] %m%n</pattern>
        </encoder>
    </appender>

    <appender name="F1" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <!-- TODO: this path should not be hard-coded -->
        <file>/var/log/puppetlabs/puppetservercompiler1/puppetserver.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>/var/log/puppetlabs/puppetservercompiler1/puppetserver-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <!-- each file should be at most 200MB, keep 90 days worth of history, but at most 1GB total-->
            <maxFileSize>200MB</maxFileSize>
            <maxHistory>90</maxHistory>
            <totalSizeCap>1GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <pattern>%d{yyyy-MM-dd'T'HH:mm:ss.SSSXXX} %-5p [%t] [%c{2}] %m%n</pattern>
        </encoder>
    </appender>

    <appender name="STATUS" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>/var/log/puppetlabs/puppetservercompiler1/puppetserver-status.log</file>
        <append>true</append>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!-- rollover daily -->
            <fileNamePattern>/var/log/puppetlabs/puppetservercompiler1/puppetserver-status-%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
            <!-- each file should be at most 200MB, keep 90 days worth of history, but at most 1GB total-->
            <maxFileSize>200MB</maxFileSize>
            <maxHistory>90</maxHistory>
            <totalSizeCap>1GB</totalSizeCap>
        </rollingPolicy>
        <encoder>
            <!-- note that this will only log the JSON message (%m) and a newline (%n)-->
            <pattern>%m%n</pattern>
        </encoder>
    </appender>

    <!-- without additivity="false", the status log messages will be sent to every other appender as well-->
    <logger name="puppetlabs.trapperkeeper.services.status.status-debug-logging" level="debug" additivity="false">
        <appender-ref ref="STATUS"/>
    </logger>

    <logger name="org.eclipse.jetty" level="INFO"/>
    <logger name="org.apache.http" level="INFO"/>
    <logger name="jruby" level="info"/>

    <root level="info">
        <!--<appender-ref ref="STDOUT"/>-->
        <!-- ${logappender} logs to console when running the foreground command -->
        <appender-ref ref="${logappender}"/>
        <appender-ref ref="F1"/>
    </root>
</configuration>

/usr/lib/systemd/system/puppetservercompiler1.service

#
# Local settings can be configured without being overwritten by package upgrades, for example
# if you want to increase puppetserver open-files-limit to 10000,
# you need to increase systemd's LimitNOFILE setting, so create a file named
# "/etc/systemd/system/puppetservercompiler1.service.d/limits.conf" containing:
#   [Service]
#   LimitNOFILE=10000
# You can confirm it worked by running systemctl daemon-reload
# then running systemctl show puppetserver | grep LimitNOFILE
#
[Unit]
Description=puppetserver compiler1 Service
After=syslog.target network.target nss-lookup.target

[Service]
Type=forking
EnvironmentFile=/etc/sysconfig/puppetservercompiler1
User=puppet
TimeoutStartSec=300
TimeoutStopSec=60
Restart=on-failure
StartLimitBurst=5
PIDFile=/run/puppetlabs/puppetservercompiler1/puppetserver.pid

# https://tickets.puppetlabs.com/browse/EZ-129
# Prior to systemd v228, TasksMax was unset by default, and unlimited. Starting in 228 a default of '512'
# was implemented. This is low enough to cause problems for certain applications. In systemd 231, the
# default was changed to be 15% of the default kernel limit. This explicitly sets TasksMax to 4915,
# which should match the default in systemd 231 and later.
# See https://github.com/systemd/systemd/issues/3211#issuecomment-233676333
TasksMax=4915

#set default privileges to -rw-r-----
UMask=027


ExecReload=/opt/puppetlabs/servercompiler1/apps/puppetserver/bin/puppetserver reload
ExecStart=/opt/puppetlabs/servercompiler1/apps/puppetserver/bin/puppetserver start
ExecStop=/opt/puppetlabs/servercompiler1/apps/puppetserver/bin/puppetserver stop

KillMode=process

SuccessExitStatus=143

StandardOutput=syslog

[Install]
WantedBy=multi-user.target

Scripts to adopt

Unluckily several scripts must be modified, too.

Mostly due to hardcoded paths.

/opt/puppetlabs/servercompiler1/apps/puppetserver/bin/puppetserver

#!/bin/bash

#set default privileges to -rw-r-----
umask 027

set -a
if [ -r "/etc/default/puppetservercompiler1" ] ; then
    . /etc/default/puppetservercompiler1
elif [ -r "/etc/sysconfig/puppetservercompiler1" ] ; then
    . /etc/sysconfig/puppetservercompiler1
elif [ `uname` == "OpenBSD" ] ; then
    JAVA_BIN=$(javaPathHelper -c puppetserver)
    JAVA_ARGS="-Xms2g -Xmx2g -Djruby.logger.class=com.puppetlabs.jruby_utils.jruby.Slf4jLogger"
    TK_ARGS=""
    USER="_puppet"
    INSTALL_DIR="/opt/puppetlabs/servercompiler1/apps/puppetserver"
    CONFIG="/etc/puppetlabs/puppetservercompiler1/conf.d"
else
    echo "You seem to be missing some important configuration files; could not find /etc/default/puppetservercompiler1 or /etc/sysconfig/puppetservercompiler1" >&2
    exit 1
fi
...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/apps/foreground

#!/usr/bin/env bash

restartfile="/opt/puppetlabs/servercompiler1/data/puppetserver/restartcounter"
cli_defaults=${INSTALL_DIR}/cli/cli-defaults.sh
...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/apps/reload

#!/usr/bin/env bash
set +e

restartfile="/opt/puppetlabs/servercompiler1/data/puppetserver/restartcounter"
reload_timeout="${RELOAD_TIMEOUT:-120}"
timeout="$reload_timeout"
realname="puppetservercompiler1"

...

initial="$(head -n 1 "$restartfile")"
pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetservercompiler1/conf.d")"
kill -HUP $pid >/dev/null 2>&1

...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/apps/start

#!/usr/bin/env bash
set +e
env

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetservercompiler1/conf.d")"

restartfile="/opt/puppetlabs/servercompiler1/data/puppetserver/restartcounter"
start_timeout="${START_TIMEOUT:-300}"

real_name="puppetservercompiler1"

...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/apps/stop

#!/usr/bin/env bash
set +e

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetservercompiler1/conf.d")"
realname="puppetservercompiler1"

...

/opt/puppetlabs/servercompiler1/apps/puppetserver/cli/cli-defaults.sh

INSTALL_DIR="/opt/puppetlabs/servercompiler1/apps/puppetserver"

if [ -n "$JRUBY_JAR" ]; then
  echo "Warning: the JRUBY_JAR setting is no longer needed and will be ignored." 1>&2
fi

CLASSPATH="${CLASSPATH}:/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/facter.jar:/opt/puppetlabs/servercompiler1/data/puppetserver/jars/*"

Adoptions to scripts from main installation

As we now have changed the way on how to determine the correct pid, we must also do this adoption at the main puppet server cli commands.

/opt/puppetlabs/server/apps/puppetserver/cli/apps/reload

...

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d")"

...

/opt/puppetlabs/server/apps/puppetserver/cli/apps/start

...

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d")"

...

/opt/puppetlabs/server/apps/puppetserver/cli/apps/stop

...

pid="$(pgrep -f "puppet-server-release.jar.* -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d")"

...

Start the stack

systemctl start puppetserver
systemctl start puppetservercompiler1

Happy hacking and success on performance tuning your Puppet server infrastructure.

Martin Alfke
ma@betadots.de

HDM Foreman integration

Martin Alfke — Thu, 25 Jan 2024 08:55:17 +0000

Analyzing Hiera Data can be very difficult.
HDM - Hiera Data manager is a web UI, which let's you analyze Hiera Data in a user-friendly way.

We now have built and released the HDM Foreman integration, which consists of two packages:

Foreman HDM Smart Proxy plugin: smart_proxy_hdm
Foreman HDM plugin: foreman_hdm

HDM Foreman Smart Proxy

Foreman needs to read data from HDM. HDM offers an API endpoint to allow remote systems to query for data and structure.
The Foreman HDM Smart Proxy plugin needs to know where it can find the HDM API endpoint and how to authorize.

HDM Foreman Plugin

The Foreman HDM plugin is an extension to the node view and adds a new tab which shows node hiera data.
Node data are fetched via HDM Smart Proxy from HDM API endpoint.

Requirements

First you need a HDM - Hiera Data manager installation. The installation can take place on an individual node or a Puppet Server.
HDM needs access to

PuppetDB to read environments, nodes and facts
Puppet Code to read hiera config file and hiera data

Besides this HDM requires to only make use of facts and trusted variables in hiera.yaml file.
Any variable which gets set during catalog compilation can not be evaluated by HDM and HDM will display these hierarchies as unresolvable.

The HDM Foreman integration needs at least Foreman 3.6 or newer. We don't build packages for older versions.
Older versions must use HDM integration from ruybgems.org.

Installation

You must install the packages from the Foreman Plugins repository and initialize the database:

dnf install -y rubygem-foreman_hdm rubygem-smart_proxy_hdm
foreman-rake db:migrate

Next you need to configure the HDM Smart Proxy:

# /etc/foreman-proxy/settings.d/hdm.yml
# HDM Smart Proxy
:enabled: https
:hdm_url: 'http://<HDM IP>:<HDM Port>'
:hdm_user: '<HDM API User Email>'
:hdm_password: '<HDM API User Password>'

HDM User and Password must be taken from an HDM API user!
You can omit these settings if you have disabled HDM user authentication.

Now the smart proxy must be restarted:

systemctl restart foreman-proxy

Screenshots

HDM Data view

Foreman Data View

Happy hacking on Puppet and Hiera data,

Martin

Puppet container version schema update

Martin Alfke — Wed, 03 Jan 2024 16:08:30 +0000

Community: we hear you!

Several people have asked to please change the version schema on puppetserver-container and puppetdb-container.

Puppet Inc version schema

Prior the repositories have been handed over from Puppet Inc to Puppet Community, the container images were using the Puppet server and PuppetDB versions, which were used inside the container.

<puppet.major>.<puppet.minor>.<puppet.patch>

Example usage:

docker run --name puppet --hostname puppet puppetlabs-puppetserver:7.2.0

Name	Description
puppet.major	Describes the contained major Puppet version (7 or 8)
puppet.minor	Describes the contained minor Puppet version
puppet.patch	Describes the contained patchlevel Puppet version

Initial voxpupuli version schema

The original version schema did not show if changes to the container image have been done.
At Voxpupuli it was decided to use a new version schema:

v<major>.<minor>.<patch>-<puppet release>

Example usage:

docker run --name puppet --hostname puppet ghcr.io/voxpupuli/container-puppetserver:v1.0.0-7

Name	Description
major	Describes the major version of the base container (Ubunutu 22.04) or incompatible changes
minor	Describes new features or refactoring with backward compatibility
patch	Describes if minor changes or bugfixes have been implemented
puppet release	Describes the contained major Puppet release (7 or 8)

At Voxpupuli Crafty we had a discussion on how to improve the version schema.
Additionally we received feedback from customers, that they were unhappy with the new version schema as it was unclear, which exact Puppet version is inside the containers.

New version schema

The new version schema has the following layout:

<puppet.major>.<puppet.minor>.<puppet.patch>-v<container.major>.<container.minor>.<container.patch>

Example usage:

docker run --name puppet --hostname puppet ghcr.io/voxpupuli/container-puppetserver:7.13.0-v1.2.0

Name	Description
puppet.major	Describes the contained major Puppet version (7 or 8)
puppet.minor	Describes the contained minor Puppet version
puppet.patch	Describes the contained patchlevel Puppet version
container.major	Describes the major version of the base container (Ubunutu 22.04) or incompatible changes
container.minor	Describes new features or refactoring with backward compatibility
container.patch	Describes if minor changes or bugfixes have been implemented

Many thanks to the community for the great feedback and ideas for improvement.

Happy hacking on Puppet in containers.

Martin Alfke

Puppet Containers are back alive

Martin Alfke — Tue, 31 Oct 2023 09:40:15 +0000

In early 2023 Puppet Inc. decided to stop working on their containers for Puppetserver and PuppetDB and offered the repositories for adoption.

At betadots GmbH we believe that there is an usecase for Puppet servers based on containers.
We also have several customers who already use Puppet in containers or are planning to migrate to a container based technology.

Therefor we decided to continue, upgrade and improve the work on the containers.

Content:

Hosting and License
Latest Updates
Future Improvements
Commercial Support

Hosting and License

To prevent license changes done by a company, we asked the Voxpupuli community to be the host for the Open Source based repositories.

This gives users and customers the benefit of being sure that the Open Source License will be kept. And also the potential number of contributors is much higher.

On mid October Tim (bastelfreak) Meusel asked Puppet's famous and well known Ben (binford2k) Ford to migrate the container repositories to Voxpupuli. They where then transferred and are now owned and maintained by Voxpupuli.

We now have two new repositories for container based Puppet infrastructure:

2 hours after migration we were able to spin up an actual version of a Puppetserver and a PuppetDB on an updated base image.
Both containers are now published under Apache-2.0 license.
betadots GmbH supports Voxpupuli with updating and maintaining the containers.

Latest Updates

For the moment we use the official Ubuntu 22.04 x86_64 base image in combination with the official Puppet Ubuntu packages released by Puppet Inc.

We are looking forward to also maintain ARM based containers, but for this we have to wait for the official Packages being finished by Puppet Inc. Puppetserver and PuppetDB are already supported on ARM, to be more precise they don't depend on an arch. But we have still to wait for the Puppet Agent to be Released on the ARM platform. The nightly builds are already there, but we wait for the final, official release. The Agent is needed in some build steps for the container, so we wait. But the build pipeline is prepared and when the packages are available it is easy to add the new arch to it.

On October 18th Robert (rwaffen) Waffen announced on Puppet Community Slack in the voxpupuli channel, that he was able to deploy a fully functional container-based Puppet setup using a (Docker) compose file.

The compose configuration is NOT hosted within the container repositories, but in the CRAFTY - Containerized Resources And Funky Tools (in) YAML repository. This will serve as the central hub for bundling all components related to the containerized Puppet infrastructure. You can think of it as a spiritual successor to Pupperware.

The containers are built to be compatible with Docker as well as other container systems such as Podman, for example. At present, they are exclusively built and tested in a Docker environment. If you encounter any issues with alternative systems, please don't hesitate to contact us. Additionally, we warmly welcome pull requests.

Future improvements

At the moment we work on a CI pipeline to automatically deploy newly built containers to docker hub and GitHub container registry. The Github part is already finished. But we might tweak some things here and there. So this is not yet production ready.

Besides this, we need support for writing a proper helm chart within the CRAFTY Repository. Anyone who can help with this would be very welcome! We aim for the containers to be used in standalone container environments and also in cluster management systems like Kubernetes (k8s).

We continue to enhance our documentation and are delighted to review and incorporate contributions.

Commercial Support

At betadots GmbH, we provide commercial support for container-based Puppet infrastructures.

Please get in touch with us if you need further information.

Happy hacking on Puppet and containers to everyone.

Martin (tuxmea) Alfke
Robert (rwaffen) Waffen
Tim (bastelfreak) Meusel