DEV Community: Tomer Figenblat

The developer's guide to Kubernetes Operators

Tomer Figenblat — Thu, 08 Feb 2024 22:54:04 +0000

Kubernetes Operators are constructed from different parts and components. This cross-referenced guide will list components you need to know to get started developing operators using the Operator Framework. You'll find a handy list of the links used at the end.

Excerpts in this guide are in Go. Tools used are part of the Operator Framework.

What is an Operator?

An Operator—aka a Kubernetes-native application—is software running and configured in a Kubernetes-based cluster, adhering to the Operator Pattern. We can write operators in any language Kubernetes has a client for. The Operator Framework offers two SDKs: Operator SDK for Go and Java Operator SDK for Java.

An operator typically describes an API and configures a manager to run controllers. Operators are deployed like any other application, using resources. As such, operators can be deployed manually or using Helm. With the Operator Framework, they are often installed using the framework's Operator Lifecycle Manager, OLM.

What is a manager?

A manager is used for creating and running our operator. We create a manager using the NewManager utility function. We use the manager's Options for configuring the various aspects of our manager instance, i.e., Scheme, Cache, Metrics, LeaderElection, HealthProbe, WebhookServer, etc.

kubeConfig := config.GetConfigOrDie()
mgr, err := ctrl.NewManager(kubeConfig, ctrl.Options{
    // manager options go here
})

Once created, we use our manager instance to create one or more controllers, a health probe, and a webhook server before using our manager's Start receiver function to run everything, which marks the start of our operator run.

// we need to include the controllers and all configurations before we start
err := mgr.Start(ctx)

What is an API?

We extend Kubernetes API using CustomResourceDefinition resources (CRD). An instance of a CRD type is called a CustomResource (CR). Resources, in general, are identified using a GroupVersionKind (GVK). The version part is reflected in our code layout as part of our API package, and we'll typically have a subpackage-per-version. From a code perspective, a CRD must implement the Object interface and have a metadata field. We use ObjectMeta as the metadata field, and TypeMeta for providing various functions related to GVK and marshaling, including the required GetObjectKind function (more on the Object interface next).

Note the marshaling markers used for the serialization of our types.

type OurCustomAPI struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata,omitempty"`
}

We also need an API List type. Used for encapsulating a list of our type instances when fetching lists. The List type implements the Object interface as well. We again use TypeMeta for the implementation of GetObjectKind. The metadata field is a ListMeta. A List type also requires an items field for encapsulating an array of instances.

type OurCustomAPIList struct {
    metav1.TypeMeta `json:",inline"`
    metav1.ListMeta `json:"metadata,omitempty"`
    Items           []OurCustomAPI `json:"items"`
}

API types will typically have a spec field for declaratively describing requirements and a status field for controllers to report back their operation status.

type OurCustomAPI struct {
    // meta objects removed for brevity
    Spec   OurCustomAPISpec   `json:"spec,omitempty"`
    Status OurCustomAPIStatus `json:"status,omitempty"`
}

type OurCustomAPISpec struct {
    // spec fields go here
}

type OurCustomAPIStatus struct {
    // status fields go here
}

Next, we must implement the DeepCopyObject receiver function to implement the Object interface fully. This is considered boilerplate and can be generated alongside other useful copy-related functions by including object generation markers in our code and running the controller-gen tool to generate the boilerplate Go code in a file named zz_generated.deepcopy.go right next to our types code.

// +kubebuilder:object:root=true
type OurCustomAPI struct {
    // fields removed for brevity
}

// +kubebuilder:object:root=true
type OurCustomAPIList struct {
    // fields removed for brevity
}

We also use CRD generation markers and CRD validation markers in our code, running the controller-gen tool to generate our CRD manifests, which will be later used for deploying our operator in the project's config/crd folder.

// +kubebuilder:object:root=true
// +kubebuilder:resource:scope=Namespaced,shortName=ocapi
type OurCustomAPI struct {
    // fields removed for brevity
    Spec OurCustomAPISpec `json:"spec,omitempty"`
}

type OurCustomAPISpec struct {
    // +kubebuilder:default=1
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=3
    Replicas *int32 `json:"replicas,omitempty"`
}

We must regenerate every time our types get any sort of modification.

As described, a CR holds the requirements in its spec and triggers one or more controllers that may reflect their operation status asynchronously in its status field. Further endpoints, called subresources, can be coupled with our custom resource.

What are subresources?

Subresources are specific simple resources that can be coupled with other resources. They are used as endpoints on top of enabled resources for performing particular operations without modifying the parent resource. In the context of CRDs, we need to be concerned with two, /scale and /status.

The /scale endpoint is used for reading and writing scaling-related data. We must add the following directive to our CRD for configuring data paths to enable scaling. Note that scaling requires us to implement a health probe to determine the health status of our operator.

We should enable the scale subresource if our operator requires scaling.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  ...
spec:
  ...
  versions:
  - name: v1alpha1
    ...
    subresources:
      scale:
        labelSelectorPath: .status.selector
        specReplicasPath: .spec.replicas
        statusReplicasPath: .status.replicas

The corresponding code will be:

type OurCustomAPISpec struct {
    Replicas *int32 `json:"replicas,omitempty"`
}

type OurCustomAPIStatus struct {
    Replicas int32    `json:"replicas,omitempty"`
    Selector string `json:"selector,omitempty"`
}

Next, the /status subresource is used for decoupling status reported by controllers from spec declarations by consumers. We add an empty object to enable it. This will effectively make our OurCustomAPIStatus a subresource.

We should enable the status subresource if we need to decouple our status from our spec. This is also considered best practice.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  ...
spec:
  ...
  versions:
  - name: v1alpha1
    ...
    subresources:
      status: {}

Once we enable the status subresource, all API requests for creating/modifying/patching instances of the related CRD will ignore the status of any object pushed through. This means that our status field must be optional. We need to include the omitempty marshaling property for it. To update the status, we use the /status endpoint, which will update only the status part. Programmatically, we can get a Client from our manager. Clients provide a Status API for creating/modifying/patching the status when it's set to be a subresource.

// when the status subresource is enabled,
// this will update only the spec and ignore the status of instanceofOurCustomAPI.
// when the status subresource is disabled,
// this will update both the spec and the status.
mgr.GetClient().Update(ctx, instanceofOurCustomAPI)

// when the status subresource is enabled,
// this will update only the status instanceofOurCustomAPI.
mgr.GetClient().Status().Update(ctx, instanceofOurCustomAPI)

Both subresources can be automatically enabled in our generated CRD manifests using the scale and status markers.

// +kubebuilder:subresource:status
// +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,selectorpath=.status.selector
type OurCustomAPI struct {
    // fields removed for brevity
}

What is a controller?

A controller, the center of the entire operator, watches the cluster's desired state for changes and attempts to reconcile the current state accordingly. We use the NewControllerManagedBy utility function to create a Builder using our manager instance, configure the API types triggering our controller, configure the Reconciler implementation invoked for every triggering event, and build our controller. We can use Predicates to fine-grain the events.

err := ctrl.NewControllerManagedBy(mgr).For(&amp;OurCustomApi{}).Complete(&amp;OurCustomReconciler{})

A Reconciler implementation exposes a Reconcile function, also known as a reconciliation loop. This function will be invoked for events based on the controller's configuration. The invocation will include the name and namespace of the triggering resource in a Request object. For every invocation, we typically fetch the latest version of the resource in question, analyze the declarative Spec, reconcile the underlying application, and update the resource's Status. The reconcile function is expected to return a Result indicating whether a requeue of the request is required and its scheduling.

When turning off requeuing, our reconciler will be invoked again only for the next triggering event.

type OurCustomReconciler struct {}

func (r *OurCustomReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // reconciliation logic goes here
    return ctrl.Result{Requeue: true, RequeueAfter: time.Minute * 5}, nil
}

When running inside a cluster, our operator will need permissions to various resources, such as our custom ones. This is achieved using RBAC. We include RBAC generation markers in our code and run the controller-gen tool to generate our Role/ClusterRole manifests, which will be later used for deploying our operator, in the project's config/rbac folder. We must regenerate as our code evolves and require RBAC modifications.

Bindings and ServiceAccounts are not generated.

// +kubebuilder:rbac:groups=group.example.com,resources=ourcustomapis,verbs=get;list;watch;create;update;patch
func (r *OurCustomReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    // removed for brevity
}

When reconciling for every event, we also need to consider Delete events. Our operator might require cleanup. For cleanup logic, we use finalizers.

What are finalizers?

Finalizers are a list of keys used as part of a mechanism for cleanup logic. When an object with a finalizer set to it is being deleted, Kubernetes will mark it for deletion by setting its DeletionTimestamp and block it from deletion until all finalizers are removed. We, the operator's developers, must add our finalizer if we need to implement any cleanup logic. We use the AddFinalizer utility function:

controllerutil.AddFinalizer(instanceofOurCustomAPI,ourOwnFinalizerString)
err := mgr.GetClient().Update(ctx, instanceofOurCustomAPI)

It's also our responsibility to remove our finalizer once the cleanup is done so the object can be deleted. We use the RemoveFinalizer utility function:

controllerutil.RemoveFinalizer(instanceofOurCustomAPI, ourOwnFinalizerString)
err := mgr.GetClient().Update(ctx, instanceofOurCustomAPI)

We use the DeletionTimestamp value and the ContainsFinalizer utility function to decide if a cleanup is in order.

if !instanceofOurCustomAPI.DeletionTimestamp.IsZero() {
    if controllerutil.ContainsFinalizer(instanceofOurCustomAPI, ourOwnFinalizerString) {
        // cleanup code goes here
    }
}

What is a scheme?

We use a Scheme reference when creating our manager to register the various types we own/work with. A Scheme acts like an object mapper, introducing types to Kubernetes. Preparing a scheme is a two-step process. First, if our code introduces new types, we must create a Builder for our GroupVersion and set our types to Register with it at runtime. A standard convention is encapsulating the Builder alongside the API, exposing its AddToScheme receiver function. This decouples the types from whomever requires them.

var (
    groupVersion = schema.GroupVersion{Group: "group.example.com", Version: "v1"}
    schemeBuilder = &amp;scheme.Builder{GroupVersion: groupVersion}
    AddToScheme = schemeBuilder.AddToScheme
)

type OurCustomAPI struct {
    // fields removed for brevity
}

type OurCustomAPIList struct {
    // fields removed for brevity
}

func init() {
    schemeBuilder.Register(&amp;OurCustomAPI{}, &amp;OurCustomAPIList{})
}

For the second step, in our operator code, just before we create our manager, we need to initiate our scheme and load types onto it using the AddToScheme function mentioned above. We create our initial scheme with the NewScheme utility function.

We use our scheme with the Scheme option when we configure our manager.

scheme := runtime.NewScheme()
err := ourapipkg.AddToScheme(scheme)

What are metrics?

Kubernetes metrics are Prometheus Metrics used for analyzing various aspects of running applications. Every manager exposes a metrics server by default. This can disabled or fine-grained. We use a preconfigured Prometheus Registry for registering metrics created using the prometheus package. We then use these metrics to report our data that, in turn, will be reflected in the metrics server.

var OurCustomCounterMetric = *prometheus.NewCounterVec(prometheus.CounterOpts{
    Name: "our_custom_counter",
    Help: "Count the thingy",
}, []string{"thingy_added_label"})

func init() {
    metrics.Registry.MustRegister(OurCustomCounterMetric)
}

Next, we can increment the metric from our code:

ourmetricspkg.OurCustomCounterMetric.WithLabelValues("labelvalue").Inc()

It's worth noting that the different components of Kubernetes report metrics that can help analyze our application.

What is a metrics server?

A metrics server is an HTTP server exposing textual data for scraping by Prometheus. We can fine-grain our server configuration when configuring our manager using the metric server's Options.

We use these options with the Metrics field when we configure our manager.

options := server.Options{BindAddress: "127.0.0.1:8080"}

By default, this server has no authentication layer. This can be mitigated with the commonly used kube-rbac-proxy as a proxy layer enforcing authentication using TLS certificates or Tokens.

What is a health probe?

Every manager can enable a HealthProbe, which means serving endpoints for checking the health of the underlying application, our operator. HealthProbe exposes two endpoints: readiness and liveliness. These are configured using our manager's receiver functions, AddHealthzCheck and AddReadyzCheck. Both take a Checker, a function that takes an HTTP Request and returns an Error. We can use the pre-built Ping checker if a custom health check is not required. Failure tolerations can be fine-grained per endpoint when designing the deployment.

Note that HealthProbe is mandatory for scaling.

err := mgr.AddHealthzCheck("healthz", healthz.Ping)
err = mgr.AddReadyzCheck("readyz", healthz.Ping)

What is a webhook server?

Our manager can expose a Webhook Server. An HTTP server serving endpoints for Dynamic Admission Controllers, AKA Admission Webhooks. These controllers are used for verifying and mutating resources before applying them to the system. There are two types of Admission Webhooks: Validating Admission Webhook and Mutating Admission Webhook. Kubernetes will invoke these webhooks for any admission of a CR. First, all the Validating Webhooks configured for the CRD will be invoked. They are expected to verify the validity of the admitted CR. Next, all the Mutating Webhooks configured for the CRD will be invoked. They are expected to mutate the CR, i.e., add labels, annotations, etc. Only after successfully invoking all webhooks, the CR will be applied to the system.

There are two approaches to implementing Admission Webhooks. The first one is by adding receiver functions on top of our API type to make it implement either or both the Validator and the Defaulter interfaces, essentially making our API type handle the logic for validation and mutation, respectively.

type OurCustomAPI struct {
    // fields removed for brevity
}

func (a *OurCustomAPI) Default() {
    // mutating logic goes here
}

func (a *OurCustomAPI) ValidateCreate() (warnings Warnings, err error) {
    // create validation logic goes here
    return nil, nil
}

func (a *OurCustomAPI) ValidateUpdate(old runtime.Object) (warnings Warnings, err error) {
    // update validation logic goes here
    return nil, nil
}

func (a *OurCustomAPI) ValidateDelete() (warnings Warnings, err error) {
    // delete validation logic goes here
    return nil, nil
}

We then use the NewWebhookManagedBy utility function to create a WebhookBuilder with our configured manager, introduce our API, and build our webhook.

err := ctrl.NewWebhookManagedBy(mgr).For(&amp;OurCustomApi{}).Complete()

As for the second implementation approach, If we prefer decoupling our webhooks from our types, we can use the CustomValidator and CustomDefaulter interfaces, supported by the WithValidator and WithDefaulter builder steps, respectively.

type OurCustomWebhook struct {}

func (w *OurCustomWebhook) Default(ctx context.Context, obj runtime.Object) error {
    // mutating logic goes here
    return nil
}

func (w *OurCustomWebhook) ValidateCreate(ctx context.Context, obj runtime.Object) (warnings Warnings, err error) {
    // create validation logic goes here
    return nil, nil
}

func (w *OurCustomWebhook) ValidateUpdate(ctx context.Context, oldObj, newObj runtime.Object) (warnings Warnings, err error) {
    // update validation logic goes here
    return nil, nil
}

func (w *OurCustomWebhook) ValidateDelete(ctx context.Context, obj runtime.Object) (warnings Warnings, err error) {
    // delete validation logic goes here
    return nil, nil
}

And create our Webhook:

ourWebhook := &amp;OurCustomWebhook{}
err := ctrl.NewWebhookManagedBy(mgr).For(&amp;OurCustomApi{}).WithValidator(ourWebhook).WithDefaulter(ourWebhook).Complete()

Next, we need to tell Kubernetes about our webhooks. This is achieved using the ValidatingWebhookConfiguration and MutatingWebhookConfiguration APIs. We can include webhook generation markers in our code and run the controller-gen tool in the project's folder to generate these manifests, which will be later used for deploying our operator, in the project's config/webhook folder. We must regenerate as our code evolves and API versions get bumped or modified.

// +kubebuilder:webhook:verbs=create;update;delete,path=/validate-group-example-com-v1beta1-ourcustomapi,mutating=false,failurePolicy=fail,groups=group.example.com,resources=ourcustomapis,versions=v1beta1,name=ourcustomapi.group.example.com,sideEffects=None,admissionReviewVersions=v1
// +kubebuilder:webhook:verbs=create;update;delete,path=/mutate-group-example-com-v1beta1-ourcustomapi,mutating=true,failurePolicy=fail,groups=group.example.com,resources=ourcustomapis,versions=v1beta1,name=ourcustomapi.group.example.com,sideEffects=None,admissionReviewVersions=v1
type OurCustomWebhook struct {}

What about versioning?

We differentiate our operator versioning from CRD versioning. For instance, our operator version 1.2.3 can support API v1beta1 and V1 while deprecating v1alpha2 and not supporting v1alpha1. Each CRD version has three properties relating to versioning. These can be fine-grained in our generated CRD manifests using CRD generation markers.

served indicates whether this version is enabled.
storage indicates the version to be used for persisting data with etcd. A mandatory one and only one version can be marked for storage.
deprecated indicates whether this version is being deprecated.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  ...
spec:
  ...
  versions:
  - name: v1alpha1
    ...
    served: false
    storage: false
  - name: v1alpha2
    ...
    served: true
    storage: false
    deprecated: true
  - name: v1beta1
    ...
    served: true
    storage: true
  - name: v1
    schema:
      ...
    served: true
    storage: false

What is Helm?

Helm is used for packaging and managing Kubernetes applications. The package format is called charts. The most notable files in a chart are the Chart.yaml file containing metadata about the chart itself, the values.yaml containing default values for our application, and the templates folder containing template YAML files and helpers for constructing our application manifests files used for deploying.

Another part of the Helm project is the Helm CLI (command-line interface), which is used for installing, upgrading, uninstalling, and managing charts on clusters. A chart installed on a cluster is called a Release. We use repositories for sharing charts. We can install a chart from a repository using the Helm CLI. We can also override the various values used for configuring the underlying application from the command line.

Helm doesn't require any operator or application installed on the target cluster. The connection between the release's components is annotation and discovery-based.

What is OLM?

The Operator Lifecycle Manager is used for managing our operator's lifecycle. OLM encapsulates operators as Managed Services, providing over-the-air updates using catalogs, a dependency model, discoverability, scalability, stability, and much more.

We first need to discuss the ClusterServiceVersion (CSV) CRs to understand OLM. The OLM operator handles these. We use a CSV to describe our operator's deployments, requirements, metadata, supported install strategies, etc. We can use the operator-sdk CLI tool and API markers to generate our CSV as a Kustomize base. By convention, we generate it in the project's config/manifests folder. We must regenerate the CSV as our code evolves and requires such modifications.

//+operator-sdk:csv:customresourcedefinitions:displayName="Our Operator API"
type OurCustomAPI struct {
    // fields removed for brevity
    Spec OurCustomAPISpec `json:"spec,omitempty"`
}

type OurCustomAPISpec struct {
    //+operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Number of replicas",xDescriptors={"urn:alm:descriptor:com.tectonic.ui:podCount"}
    Replicas int `json:"replicas"`
}

The next OLM component we'll cover is the Bundle. After generating our CSV, we'll include additional supported resources in the config folder based on the common structure, use kustomize, and pipe the result into the operator-sdk cli tool to generate our Bundle manifests. We use a bundle per operator version. Bundles are container images built from scratch, adhering to a specific folder layout:

The/manifests folder contains our operator's CSV mentioned above and other supported resources required for our operator to run, i.e., CRDs, RBAC, etc.
The /metadata folder contains a file called annotations.yaml with all the annotations used for configuring our Bundle, i.e. package and channel data. We can optionally include dependencies.yaml file to leverage OLM's dependency resolution mechanism.
The (optional) /tests/scorecard folder contains a config.yaml file, used for statically validating and testing our operator.

After building and pushing our Bundle image to an accessible registry, we can potentially install or upgrade the Bundle directly from the image using the operator-sdk cli tool. But the next logical step would be to include our Bundle in a Catalog. The Catalog is a container image built from OPM's Image serving the Catalogs configured in its /configs folder. The configs are YAML files describing multiple schemas. The olm.package schema describes a package. The olm.bundle schema describes bundles for a package. Each Bundle represents a specific version of our operator and targets a specific Bundle image. Next, the olm.channel schema is used for associating package bundles to a channel, i.e., selecting bundles for the stable and alpha channels. See example. We use the opm cli tool to generate the various components.

One of the more known open source catalogs is OperatorHub.io

After building and pushing our Catalog image to an accessible registry, consumers can create a CatalogSource targeting our image and install packages from our Catalog. Consumers create OperatorGroups, setting the allowed permission for installed operators. And Subscriptions for requesting a package to be installed by OLM. This will create an InstallPlan describing the installation of the package. Once approved, manually or automatically, the underlying operator's Bundle selected for the target Channel of the desired Package will be installed based on its CSV.

What is Kustomize?

Kustomize helps create and manage resource files template-free by merging overlays on top of bases and introducing various built-in and customizable transformers and generators. We use a kustomize.yaml file to declaratively orchestrate our manifests construction and the kustomize cli tool to build our manifests. Throughout this guide, we noticed that the generated file target is usually config/X in our project's root, we typically design our kustomization on top of the config folder.

Quick links

Development tools

controller-gen generates CRD, RBAC, and Webhook manifests, as well as DeepCopy boilerplate functions.
operator-sdk generates CSV and Bundles for OLM.
opm composes Catalogs for OLM.
kustomize constructs Kubernetes manifests using overlays, transformers, and generators.
helm deployes and manages Helm Chart Releases.

Development toolbox

GetConfigOrDie is used for fetching the configuration for accessing the running server.
NewScheme is used for creating a new scheme for introducing types to Kubernetes.
NewManager is used for creating a new manager for creating controllers and running the operator.
NewControllerManagedBy is used for configuring a new controller with a manager.
NewWebhookManagedBy is used for configuring a new admission webhook with a manager.
Ping Health Checker is a useful health probe checker.
Prometheus Registry Kubernetes' pre-configured Prometheus registry.
Unstructured Type provides the Unstructured type.

Useful packages

Controller Utils Package provides various utils for working with controllers.
Meta Package contains valuable functions for working with objects meta.
Prometheus Package for creating metrics.
Errors Package provides useful utility functions for working with errors.
CLI Package provides utility functions for creating Cobra commands (common approach for creating operators).
Zap Package provides utility functions for createing loggers.
Wait Package provides tools for listening for Conditions.
JSON Package provides utility function for marsharling and unmarshaling JSON.
Rand Package provides tools related to randomization.
Version Package provides utility function for inspecting versions.
Yaml Package provides utility function for marsharling and unmarshaling Yaml.

Common interfaces

Code markers

How to distribute workloads using Open Cluster Management

Tomer Figenblat — Fri, 26 Jan 2024 14:56:55 +0000

Open Cluster Management (OCM) was accepted to the Cloud Native Computing Foundation (CNCF) in late 2021 and is currently at the Sandbox project maturity level. OCM is a community-driven project focused on multicluster and multicloud scenarios for Kubernetes applications. This article shows how to bootstrap Open Cluster Management and handle work distribution using ManifestWork. We also discuss several ways to select clusters for various tasks using the ManagedClusterSet and Placement resources.

Basic Open Cluster Management

Open Cluster Management is based on a hub-spoke architecture. In this design, a single hub cluster prescribes prescriptions, and one or more spoke clusters act upon these prescriptions. In Open Cluster Management, spoke clusters are called managed clusters. The component running on the hub cluster is the cluster manager. The component running on a managed cluster, is the klusterlet agent.

OCM requires one-sided communication. From the managed clusters to the hub cluster. The communication to the hub is handled by two APIs:

Registration: For joining managed clusters to the hub cluster and manage their lifecycle.
Work: For relaying workloads in the form of prescriptions prescribed on the hub cluster and reconciled on the managed clusters.

OCM resources are easily managed using a command-line interface (CLI) named clusteradm.

A number of resources help you select the clusters used in your application. We will look at all these resources in this article:

ManagedClusterSets collect ManagedClusters into groups.
Placements let you select clusters from a ManagedClusterSet.
To select clusters for a placement, you can use labels, ClusterClaims, taints and tolerations, and prioritizers.

As a simple, basic use case for placements, say you have two ManagedClusterSets, one for all the clusters in Israel and one for all the clusters in Canada. You can use placements to cherry-pick from these sets the managed clusters that are suited for testing or production.

Setting up the example environment

To get started, you'll need to install clusteradm and kubectl and start up three Kubernetes clusters. To simplify cluster administration, this article starts up three kind clusters with the following names and purposes:

kind-fedora1 runs the hub cluster.
kind-rhel1 runs one managed cluster.
kind-qnap1 runs another managed cluster.

Bootstrapping OCM

The bootstrapping task consists of initializing the hub cluster and joining the managed clusters to it. OCM registration involves a double opt-in handshake initiated by the managed clusters and accepted by the hub cluster. At any point in time, the connection can be ended by either party.

Initializing the hub cluster

Run the following command to initialize the hub cluster. Output from the clusteradm command is filtered by the grep command and assigned to the joinCommand variable:

$ joinCommand=$(clusteradm init --context kind-fedora1 --wait | grep clusteradm)

Note: This command includes the deployment of the cluster manager, so it might take a couple of minutes.

You can verify that the cluster manager is running by checking for pods in the designated open-cluster-management-hub and open-cluster-management namespaces.

Joining and accepting the managed clusters

Join each managed cluster to the hub by injecting the cluster name into the stored joinCommand variable and running the command. This command should be run for every managed cluster. In our example, we have two clusters to manage. As mentioned earlier, registration is a double opt-in handshake, so every join request needs to get accepted by the hub cluster through a clusteradm command:

$ eval $(echo "$joinCommand --context kind-rhel1 --insecure-skip-tls-verify --wait" | sed 's/<cluster_name>/kind-rhel1/g' -)
$ eval $(echo "$joinCommand --context kind-qnap1 --insecure-skip-tls-verify --wait" | sed 's/<cluster_name>/kind-qnap1/g' -)
$ clusteradm --context kind-fedora1 accept --clusters kind-rhel1,kind-qnap1 --wait

Note: These commands deploy the klusterlet agents and initialize the registration, so they might take a couple of minutes.

You can verify that the klusterlet agent is running by checking for pods in the designated open-cluster-management-agent and open-cluster-management namespaces for every managed cluster.

For every managed cluster initiating a join request, the registration api creates a cluster-scoped ManagedCluster resource on the hub cluster with the specification and status for the associated managed cluster.

Excerpts from the resource for one of the clusters in our example, kind-rhel1, follow. Note the status object, which holds valuable information about the cluster:

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: kind-rhel1
  ...
spec:
  hubAcceptsClient: true
  leaseDurationSeconds: 60
  ...
status:
  allocatable:
    cpu: "4"
    ephemeral-storage: 71645Mi
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8012724Ki
    pods: "110"
  capacity:
    cpu: "4"
    ephemeral-storage: 71645Mi
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    memory: 8012724Ki
    pods: "110"
  conditions:
  - lastTransitionTime: "2022-12-21T10:23:47Z"
    message: Accepted by hub cluster admin
    reason: HubClusterAdminAccepted
    status: "True"
    type: HubAcceptedManagedCluster
  - lastTransitionTime: "2022-12-21T10:23:47Z"
    message: Managed cluster joined
    reason: ManagedClusterJoined
    status: "True"
    type: ManagedClusterJoined
  - lastTransitionTime: "2022-12-21T10:23:47Z"
    message: Managed cluster is available
    reason: ManagedClusterAvailable
    status: "True"
    type: ManagedClusterConditionAvailable
  version:
    kubernetes: v1.25.3

Workload distribution across managed clusters

After a successful registration, the hub cluster creates a designated namespace for every joined managed cluster. These namespaces are called cluster namespaces and form the targets for workload distributions. In our example, we expect two namespaces named kind-qnap1 and kind-rhel1.

It's important to note that cluster namespaces are used for only workload distributions and certain other activities related to the managed cluster. Anything else, such as placement resources (discussed later in this article) and subscriptions (to be discussed in a future article), go into your application's namespace.

To distribute workloads across managed clusters, apply a ManifestWork resource describing your workload in the cluster namespace. The Klusterlet Agents periodically check for ManifestWorks in their designated namespaces, reconcile themselves, and report back with the reconciliation status.

Here's a simple example of a ManifestWork including a namespace and a simple deployment. When applied to a cluster namespace, the associated managed cluster will apply the workload in an orderly fashion:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  namespace: <target managed cluster>
  name: hello-work-demo
spec:
  workload:
    manifests:
      - apiVersion: v1
        kind: Namespace
        metadata:
          name: hello-workload
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: hello
          namespace: hello-workload
        spec:
          selector:
            matchLabels:
              app: hello
          template:
            metadata:
              labels:
                app: hello
            spec:
              containers:
                - name: hello
                  image: quay.io/asmacdo/busybox
                  command:
                    ["sh", "-c", 'echo "Hello, Kubernetes!" &amp;&amp; sleep 3600']

You can verify that the distribution took place by looking for the hello deployment in the hello-workload namespace in each cluster you intended to deploy to.

After deploying this ManifestWork, the klusterlet agent of the associated managed cluster creates the necessary resources and reports back. Excerpts from a status report for the previous ManifestWork follow:

apiVersion: work.open-cluster-management.io/v1
kind: ManifestWork
metadata:
  ...
spec:
  ...
status:
  conditions:
  - lastTransitionTime: "2022-12-21T11:15:35Z"
    message: All resources are available
    observedGeneration: 1
    reason: ResourcesAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2022-12-21T11:15:35Z"
    message: Apply manifest work complete
    observedGeneration: 1
    reason: AppliedManifestWorkComplete
    status: "True"
    type: Applied
  resourceStatus:
    manifests:
    - conditions:
      - lastTransitionTime: "2022-12-21T11:15:35Z"
        message: Apply manifest complete
        reason: AppliedManifestComplete
        status: "True"
        type: Applied
      - lastTransitionTime: "2022-12-21T11:15:35Z"
        message: Resource is available
        reason: ResourceAvailable
        status: "True"
        type: Available
      - lastTransitionTime: "2022-12-21T11:15:35Z"
        message: ""
        reason: NoStatusFeedbackSynced
        status: "True"
        type: StatusFeedbackSynced
      resourceMeta:
        group: ""
        kind: Namespace
        name: hello-workload
        namespace: ""
        ordinal: 0
        resource: namespaces
        version: v1
      statusFeedback: {}
    - conditions:
      - lastTransitionTime: "2022-12-21T11:15:35Z"
        message: Apply manifest complete
        reason: AppliedManifestComplete
        status: "True"
        type: Applied
      - lastTransitionTime: "2022-12-21T11:15:35Z"
        message: Resource is available
        reason: ResourceAvailable
        status: "True"
        type: Available
      - lastTransitionTime: "2022-12-21T11:15:35Z"
        message: ""
        reason: NoStatusFeedbackSynced
        status: "True"
        type: StatusFeedbackSynced
      resourceMeta:
        group: apps
        kind: Deployment
        name: hello
        namespace: hello-workload
        ordinal: 1
        resource: deployments
        version: v1
      statusFeedback: {}

For every ManifestWork resource identified on the hub cluster, the klusterlet agent creates a cluster-scoped AppliedManifestWork resource on the managed cluster. This resource serves as the owner and the status reporter for the workload. Excerpts from the AppliedManifestWork for the previous ManifestWork follow:

apiVersion: work.open-cluster-management.io/v1
kind: AppliedManifestWork
metadata:
  ...
spec:
  ...
  manifestWorkName: hello-work-demo
status:
  appliedResources:
  - group: ""
    name: hello-workload
    namespace: ""
    resource: namespaces
    version: v1
    ....
  - group: apps
    name: hello
    namespace: hello-workload
    resource: deployments
    version: v1
    ...

Grouping managed clusters

As explained earlier, the hub cluster creates a cluster-scoped ManagedCluster to represent each managed cluster joined. You can group multiple ManagedClusters using the cluster-scoped ManagedClusterSet.

At the start, there are two pre-existing ManagedClustetSets: The default one, which includes every newly joined ManagedCluster, and the global one that includes all ManagedClusters:

$ clusteradm --context kind-fedora1 get clustersets

<ManagedClusterSet>
└── <default>
│   ├── <BoundNamespace>
│   ├── <Status> 2 ManagedClusters selected
└── <global>
    └── <Status> 2 ManagedClusters selected
    └── <BoundNamespace>

Now add your own set:

$ clusteradm --context kind-fedora1 create clusterset managed-clusters-region-a

Next, configure both of your clusters as members. The following command overwrites the designated label for ManagedClusters custom resources:

$ clusteradm --context kind-fedora1 clusterset set managed-clusters-region-a --clusters kind-rhel1,kind-qnap1

When you set your clusters as members of your cluster set, they are removed from the pre-existing default set.

As stated earlier, the ManagedClusterSet resource is cluster-scoped. When you write your application using the cluster set, you have to bind it into your application's namespace, which you can do using clusteradm.

The following command creates a namespace-scoped ManagedClusterSetBinding custom resource in your application namespace:

$ clusteradm --context kind-fedora1 clusterset bind managed-clusters-region-a --namespace our-application-ns

You can verify through the following command that your clusters have moved from the default set to your new set, and that your set is bound to your application namespace:

$ clusteradm --context kind-fedora1 get clustersets

<ManagedClusterSet>
└── <default>
│   ├── <BoundNamespace>
│   ├── <Status> No ManagedCluster selected
└── <global>
│   ├── <Status> 2 ManagedClusters selected
│   ├── <BoundNamespace>
└── <managed-clusters-region-a>
    └── <BoundNamespace> our-application-ns
    └── <Status> 2 ManagedClusters selected

Selecting clusters from the set

With your ManagedClusterSet bound to your application namespace, you can create a Placement to dynamically select clusters from the set. The following subsections show several ways to select clusters, fetch the selected cluster list, and prioritize managed clusters.

Using labels to select clusters

You can select clusters for a placement using labels. The following configuration tells your placement which labels to look for in ManagedClusters within the ManagedClusterSets configured:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: our-label-placement
  namespace: our-application-ns
spec:
  numberOfClusters: 1
  clusterSets:
    - managed-clusters-region-a
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchLabels:
            our-custom-label: "include-me-4-tests"

Because you haven't yet labeled any of your ManagedClusters yet, your placement will not find any clusters:

$ kubectl --context kind-fedora1 get placement -n our-application-ns

NAME                  SUCCEEDED   REASON                    SELECTEDCLUSTERS
our-label-placement   False       NoManagedClusterMatched   0

So add the appropriate label on one of your ManagedClusters, i.e. kind-rhel1:

$ kubectl --context kind-fedora1 label managedcluster kind-rhel1 our-custom-label="include-me-4-tests"

Your placement should now pick this up:

$ kubectl --context kind-fedora1 get placement -n our-application-ns

NAME                  SUCCEEDED   REASON                  SELECTEDCLUSTERS
our-label-placement   True        AllDecisionsScheduled   1

Using ClusterClaims to select clusters

ClusterClaims address two concerns with the use of labels for placement. The first is that, although labels are useful, their overuse makes them error-prone. The second concern is that, in our case, the labels are added for resources on the hub cluster, requiring the cluster administrator or another permitted user with access.

With ClusterClaims, the selection of clusters can be delegated to the managed clusters. ClusterClaims are custom resources applied on the managed cluster. Their content is propagated to the hub as a status for their associated ManagedCluster resources. The cluster administrators for a managed cluster can decide, for instance, which of their clusters are used for tests and which are used for production by simply applying this agreed-upon custom resource.

Clusterclaims can also be used in conjunction with labels to fine-grain your selection.

Apply the following YAML in one of your managed clusters. This example uses kind-qnap1:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: ClusterClaim
metadata:
  name: our-custom-clusterclaim
spec:
  value: include-me-for-tests

Propagation can be verified on the associated ManagedCluster resource on the hub cluster:

apiVersion: cluster.open-cluster-management.io/v1
kind: ManagedCluster
metadata:
  name: kind-qnap1
  ...
spec:
  hubAcceptsClient: true
  ...
status:
  allocatable:
    ...
  capacity:
    ...
  clusterClaims:
  - name: our-custom-clusterclaim
    value: include-me-for-tests
  conditions:
  ...
  version:
    kubernetes: v1.25.3

Now you can create a placement based on your ClusterClaim:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: our-clusterclaim-placement
  namespace: our-application-ns
spec:
  numberOfClusters: 1
  clusterSets:
    - managed-clusters-region-a
  predicates:
    - requiredClusterSelector:
        claimSelector:
          matchExpressions:
            - key: our-custom-clusterclaim
              operator: In
              values:
                - include-me-for-tests

And verify the placement selected the claimed ManagedCluster:

$ kubectl --context kind-fedora1 get placement -n our-application-ns

NAME                         SUCCEEDED   REASON                  SELECTEDCLUSTERS
our-clusterclaim-placement   True        AllDecisionsScheduled   1
our-label-placement          True        AllDecisionsScheduled   1

Using taints and tolerations for the selection

Taints and tolerations help you filter out unhealthy or otherwise not-ready clusters.

Taints are properties of ManagedCluster resources. The following command adds a taint to make your placement deselect your kind-qnap1 cluster. The command also removes existing taints from this ManagedCluster, so be careful when executing such commands:

$ kubectl --context kind-fedora1 patch managedcluster kind-qnap1 --type='json' -p='[{"op": "add", "path": "/spec/taints", "value": [{"effect": "NoSelect", "key": "our-custom-taint-key", "value": "our-custom-taint-value" }] }]'

To verify that the taint was added, execute:

$ kubectl --context kind-fedora1 get managedcluster kind-qnap1 -o jsonpath='{.spec.taints[*]'}

{"effect":"NoSelect","key":"our-custom-taint-key","timeAdded":"2022-12-22T15:38:47Z","value":"our-custom-taint-value"}

Now verify that the relevant placement has deselected your cluster based on the NoSelect effect:

$ kubectl --context kind-fedora1 get placement -n our-application-ns our-clusterclaim-placement

NAME                         SUCCEEDED   REASON                    SELECTEDCLUSTERS
our-clusterclaim-placement   False       NoManagedClusterMatched   0

A toleration overrides taints. So your next experiment is to make your placement ignore the previous taint using a toleration. Again, be careful when executing commands like the following because it removes any existing tolerations from the placement:

$ kubectl --context kind-fedora1 patch placement -n our-application-ns our-clusterclaim-placement --type='json' -p='[{"op": "add", "path": "/spec/tolerations", "value": [{"key": "our-custom-taint-key", "value": "our-custom-taint-value", "operator": Equal }] }]'

Verify that the toleration was added:

$ kubectl --context kind-fedora1 get placement -n our-application-ns our-clusterclaim-placement -o jsonpath='{.spec.tolerations[*]'}

"key":"our-custom-taint-key","operator":"Equal","value":"our-custom-taint-value"}

And verify that the relevant placement has reselected your cluster:

$ kubectl --context kind-fedora1 get placement -n our-application-ns our-clusterclaim-placement

NAME                         SUCCEEDED   REASON                  SELECTEDCLUSTERS
our-clusterclaim-placement   True        AllDecisionsScheduled   1

Two taints are automatically created by the system:

cluster.open-cluster-management.io/unavailable
cluster.open-cluster-management.io/unreachable

You can't manually modify these taints, but you can add tolerations to override them. You can even issue temporary tolerations that last a specified number of TolerationSeconds, as described in the placement documentation.

Fetching the selected cluster list

As long as a placement has selected at least one cluster, the system creates PlacementDecision resources listing the selected clusters:

$ kubectl --context kind-fedora1 get placementdecisions -n our-application-ns

NAME                                    AGE
our-clusterclaim-placement-decision-1   10m
our-label-placement-decision-1          22m

A PlacementDecision has the same namespace and name as its placement counterpart.

Your PlacementDecision for the placement selected by label should display your labeled ManagedCluster:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: PlacementDecision
metadata:
  name: our-label-placement-decision-1
  namespace: our-application-ns
  ownerReferences:
  - apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    name: our-label-placement
    ...
  ...
status:
  decisions:
  - clusterName: kind-rhel1
    reason: ""

Similarly, your PlacementDecision for the placement selected by ClusterClaim should display your claimed ManagedCluster:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: PlacementDecision
metadata:
  name: our-clusterclaim-placement-decision-1
  namespace: our-application-ns
  ownerReferences:
  - apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    name: our-clusterclaim-placement
    ...
  ...
status:
  decisions:
  - clusterName: kind-qnap1
    reason: ""

Prioritizing clusters for selection

Prioritizers tell placements to prefer some clusters based on built-in ScoreCoordinates. You can also extend prioritization by using an AddOnPlacementScore.

The following settings configure your placement to use prioritization and sort your clusters based on the allocated memory and CPU capacity (as viewed in the cluster status). Each coordinate is assigned a different weight for the selection:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: our-clusterclaim-placement
  namespace: our-application-ns
spec:
  numberOfClusters: 2
  clusterSets:
    ...
  prioritizerPolicy:
    mode: Exact
    configurations:
      - scoreCoordinate:
          builtIn: ResourceAllocatableMemory
        weight: 2
      - scoreCoordinate:
          builtIn: ResourceAllocatableCPU
        weight: 3
  predicates:
    - requiredClusterSelector:
        ...

To extend the built-in score coordinates, use a designated AddOnPlacementScore in your application namespace:

apiVersion: cluster.open-cluster-management.io/v1alpha1
kind: AddOnPlacementScore
metadata:
  name: our-addon-placement-score
  namespace: our-application-ns
status:
  conditions:
  - lastTransitionTime: "2021-10-28T08:31:39Z"
    message: AddOnPlacementScore updated successfully
    reason: AddOnPlacementScoreUpdated
    status: "True"
    type: AddOnPlacementScoreUpdated
  validUntil: "2021-10-29T18:31:39Z"
  scores:
  - name: "our-custom-score-a"
    value: 66
  - name: "our-custom-score-b"
    value: 55

Now, you can modify your placement and add score coordinates referencing your custom addon scores:

apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: our-clusterclaim-placement
  namespace: our-application-ns
spec:
  numberOfClusters: 2
  clusterSets:
    ...
  prioritizerPolicy:
    mode: Exact
    configurations:
      - scoreCoordinate:
          builtIn: ResourceAllocatableMemory
        weight: 2
      - scoreCoordinate:
          builtIn: ResourceAllocatableCPU
        weight: 3
      - scoreCoordinate:
          builtIn: AddOn
          addOn:
            resourceName: our-addon-placement-score
            scoreName: our-custom-score-a
        weight: 1
      - scoreCoordinate:
          builtIn: AddOn
          addOn:
            resourceName: our-addon-placement-score
            scoreName: our-custom-score-b
        weight: 4
  predicates:
    - requiredClusterSelector:
        ...

Summarizing OCM basics

This article introduced Open Cluster Management and explained how to:

Bootstrap OCM on hub and spoke (managed) clusters.
Deploy a workload across managed clusters.
Group managed clusters into sets.
Select clusters from cluster sets.
Customize placement scheduling.

Upcoming articles will cover various features, addons, frameworks, and integrations related to OCM. These components make use of the infrastructure concerning work distributions and placements described in this article. In the meantime, you can learn about how to prevent computer overload with remote kind clusters. I want to thank you for taking the time to read this article, and I hope you got something out of it. Feel free to comment below if you have questions. We welcome your feedback.

How to prevent computer overload with remote kind clusters

Tomer Figenblat — Fri, 26 Jan 2024 14:09:45 +0000

Kubernetes can require a lot of resources, which can overload a developer's laptop. This article shows you how to use a set of tools—including kind, kubeconfig, and Podman or Docker—to spread your files around remote systems in support of your local development work.

Why I researched tools to prevent computer overload

Lately, I've been working a lot with Open Cluster Management, a community-driven project focused on multicluster and multi-cloud scenarios for Kubenetes applications.

The Open Cluster Management topology is hub-spoke based, calling for one hub cluster and at least one spoke cluster. That means that, throughout my work, I needed at least two clusters running simultaneously.

The quickest way to get two clusters up and running for development purposes is to use kind, a Kubernetes management tool. With kind, you can easily spin up Kubernetes clusters running in containers on your local computer.

One of my tasks included working with Prometheus, so I needed multiple clusters running the Prometheus operator plus the required operators for running Open Cluster Management, including the Application Lifecycle Manager addon. The load on my local computer was, eventually, too much for it to handle, and it eventually stopped cooperating with me.

To work around this bottleneck, I decided to spread my kind clusters to multiple computers around the office, import their kubeconfig files to my local computer, and continue to work as if the clusters were local.

Each of the remote computers needed to install kind as well as a container engine. To manage the containers I used Podman, but Docker should do just as well.

For access to remote clusters, SSH is usually preferable, but any manner of getting access to them should suffice. After spinning up a kind cluster and exporting the relevant kubeconfig file, you will no longer need access to the remote clusters, with the exception of the designated port 6443 for access to the Kubernetes API server.

How to set up kind remote clusters

The remote computer's IP address I use in the following examples is 192.168.1.102.

Assuming you have SSH, connect to the remote computer as follows:

$ ssh 192.168.1.102

Create a custom kind cluster using the following command. Note the networking property, which is required to make your cluster's API server listen on the right address so you can reach it from your local computer on the same network:

$ kind create cluster --config=- &lt;&lt; EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: remote-cluster1
nodes:
- role: control-plane
networking:
  apiServerAddress: "192.168.1.102"
  apiServerPort: 6443
EOF

Now, still on the remote computer, use kind to export the cluster configuration into a file of your choice; the following command names the file remote_kube_config:

$ kind get kubeconfig --name remote-cluster1 &gt; ~/remote_kube_config

Now go back to your local computer and copy your current configuration into a file that I'll call local_kube_config. This file can also serve as a backup:

$ cp ~/.kube/config ~/local_kube_config

Then run the following command to copy the remote configuration to your local computer over SSH:

$ scp 192.168.1.102:~/remote_kube_config ~

Now merge the two configuration files. Note that if you have many remote clusters, you can include multiple configuration files in the following command:

$ KUBECONFIG="${HOME}/local_kube_config:${HOME}/remote_kube_config" kubectl config view --flatten &gt; ~/.kube/config

Verify access to your remote kind cluster from your local computer:

$ kubectl get nodes --context kind-remote-cluster1

NAME                            STATUS   ROLES           AGE   VERSION
remote-cluster1-control-plane   Ready    control-plane   19m   v1.25.3

The output shows that you have access to the cluster.

Bonus: Loading images to remote clusters

When you need to load images from your local storage to a local kind cluster, you can take advantage of the following useful command:

$ kind load docker-image &lt;image-registry&gt;/&lt;image-owner&gt;/&lt;image-name&gt;:&lt;image-tag&gt; --name local-cluster

But when working with remote clusters, this process gets tricky. In the previous section, you made kubectl aware of your remote cluster by merging its kubeconfig configuration, but your local instance of kind has no idea who remote-cluster1 is.

Images can be loaded only to local kind clusters. This means that to load a file onto a remote computer, you need to get your image into your remote computer's storage and load it from there.

To do that, first archive your image:

$ podman save &lt;image-registry&gt;/&lt;image-owner&gt;/&lt;image-name&gt;:&lt;image-tag&gt; -o archive-file-name

Then copy the archive to your remote computer:

$ scp archive-file-name 192.168.1.102:~

Connect using SSH to the remote computer:

$ ssh 192.168.1.102

And load the archive as an image to your kind cluster:

$ kind load image-archive archive-file-name --name remote-cluster1

Tools that simplify Kubernetes

For more information on tools that simplify work with Kubernetes, visit Red Hat's Developer Tools page. Please check out my next article, How to distribute workloads using Open Cluster Management. Feel free to comment below if you have questions. We welcome your feedback. Have a great day, and keep up the good work!

Protect secrets in Git with the clean/smudge filter

Tomer Figenblat — Fri, 26 Jan 2024 02:39:18 +0000

When working on public Git repositories, you need to pay close attention so that you don't accidentally push secret information such as tokens, private server addresses, personal email addresses, and the like. One of the tools that can help you is Git's clean/smudge filter.

Clean and smudge your Git repository

The clean/smudge filter is quite simple to use. Create a filter driver with two commands—clean and smudge—and then apply the filter per record in the .gitattributes file.

This makes the filter stand between the working directory and the staging area for the specific .gitattributes record. When adding content from the working directory to the staging area with git add, files that match the .gitattributes record will go through the clean filter. When pulling content back into the working directory with git pull, those same files will go through the smudge filter.

Create the filter driver

Let's create a filter driver named cleanPass that uses sed expressions to replace the value secretpassword with the value hiddenpassword and vice versa:

# mac users should use gsed instead of sed
git config filter.cleanPass.clean "sed -e 's/secretpassword/hiddenpassword/g'"
git config filter.cleanPass.smudge "sed -e 's/hiddenpassword/secretpassword/g'"

Tip: Add --global to the config command to make the filter driver available globally to all repositories at ~/.gitconfig instead of locally at .git/config.

As an example, let's apply the cleanPass filter driver to the JSON file type. In the repository's .gitattributes, look for the JSON file type record and modify it. The result should look something like this, depending on the original configuration:

*.json text eol=lf filter=cleanPass

Tip: To avoid pushing, this can be done in .git/info/attributes, which takes precedence over .gitattributes. Just be sure not to mess up any of the original record configurations like eol.

From now on, every new or modified JSON file in the repository will go through the cleanPass filter. Figure 1 provides an animated look at what happens next.

One filter to rule them all

That was cool, but using this filter can get quite cumbersome at times. For instance, on a recent task, I worked with multiple repositories using multiple (but identical) services from my team's Red Hat OpenShift lab. To add to the complexity, I usually work from two different computers—my desktop at my office and my laptop at home.

That all means that I needed to create multiple filters (or one complex one) and apply them to multiple file types, for multiple repositories, on multiple computers. That gave me a headache, especially when the fact hit me that this workflow will probably reoccur in future tasks.

Eventually, I decided to create a very simple script that handles multiple value replacements in both directions, save it on each of my computers, and configure it as a global filter.

Configure a global filter

Adding values to be replaced in this script is super easy, and attaching it to .gitattributes records, as seen above, is quite simple as well.

First, create a script in a path available to all repositories. For instance, you could put it in your home directory at ~/scripts/git-smudge-clean-filter.sh:

#!/bin/bash

declare -A mapArr

mapArr["my-work-private-server.mywork.com"]="&lt;reducted-work-server&gt;"
mapArr["my-personal-private-server.myowndomain.org"]="&lt;reducted-personal-server&gt;"
mapArr["A*&amp;#QAADDA(77##F"]="super-secret-token"
mapArr["oops@mypersonal.email"]="support@correct.email"

# mac users should use gsed instead of sed
sedcmd="sed"
if [[ "$1" == "clean" ]]; then
  for key in ${!mapArr[@]}; do
    sedcmd+=" -e \"s/${key}/${mapArr[${key}]}/g\""
  done  
elif [[ "$1" == "smudge" ]]; then
  for key in ${!mapArr[@]}; do
    sedcmd+=" -e \"s/${mapArr[${key}]}/${key}/g\""
  done  
else  
  echo "use smudge/clean as the first argument"
  exit 1
fi

eval $sedcmd

Add as many pairs as you like to mapArr.

Note: Per the nature of Git's filter driver, you can't use the same key or value more than once.

Next, create the filter driver globally (note the script argument):

git config --global filter.reductScript.smudge "~/scripts/git-smudge-clean-filter.sh smudge"
git config --global filter.reductScript.clean "~/scripts/git-smudge-clean-filter.sh clean"

Now, for every .gitattributes record to which you apply the filter=reductScript configuration in any repository, every file matching this record will go through the script, and every value specified in mapArr will be reducted (or returned).

Conclusion

That's all there is to it. I hope you find this useful. We'll cover other Git tips in future articles.

In the meantime, check out the following Git resources:

Testing Java with JUnit Pioneer

Tomer Figenblat — Mon, 24 May 2021 10:32:11 +0000

Making test code readable with JUnit Pioneer

Browsing a project's test cases is a common approach to better understanding the codebase. Making our tests readable has a tremendous impact on this approach's feasibility. For that, JUnit Pioneer is a mandatory tool in your toolset!

This article won't cover everything included with JUnit Pioneer. It's all pretty well documented.

Let's take the following example:

class App {
  private static final Set<String> acceptableAnswers = Set.of("yes", "sure");

  public static void main(final String... args) {
    System.out.println("Hello, do you love Java?");
    verifyResponse(new Scanner(System.in));
    System.out.println("Me too!");
  }

  private static void verifyResponse(final Scanner scanner) {
    if (!acceptableAnswers.contains(scanner.next().toLowerCase())) {
      System.out.println("""
          Hmmm... I'm not sure I understand.
          Please reply with 'yes' or 'sure'.
          Do you love Java?""");
      verifyResponse(scanner);
    }
  }
}

This is pretty straightforward, executing the main method will ask: "Hello, do you love Java?" and will not accept any answers other than "yes" or "sure".

The common approach for testing code that takes data from the System's in stream and prints it to its out stream is to hijack the streams:

class Manual_Hijack_System_Streams_Test {
  private InputStream origIn;
  private PrintStream origOut;
  private OutputStream output;

  @BeforeEach
  void initialize() {
    origIn = System.in;
    origOut = System.out;
    output = new ByteArrayOutputStream();
    System.setOut(new PrintStream(output));
  }

  @AfterEach
  void cleanup() {
    System.setIn(origIn);
    System.setOut(origOut);
  }

  @ParameterizedTest
  @ValueSource(strings = {"yes", "sure"})
  void verify_replying_with_an_acceptable_answer(final String answer) {
    System.setIn(inputStreamOf(answer));
    App.main(new String[0]);

    assertArrayEquals(
      new String[] {"Hello, do you love Java?", "Me too!"},
      output.toString().split(System.lineSeparator()));
  }

  @Test
  void verify_replying_with_a_wrong_and_an_acceptable_answer() {
    System.setIn(inputStreamOf(collectAnswers("no", "yes")));
    App.main(new String[0]);

    assertArrayEquals(
      new String[] {
        "Hello, do you love Java?",
        """
            Hmmm... I'm not sure I understand.
            Please reply with 'yes' or 'sure'.
            Do you love Java?""",
        "Me too!"},
      output.toString().split(System.lineSeparator()));
  }

  private InputStream inputStreamOf(final String input) {
    return new ByteArrayInputStream(input.getBytes(UTF_8));
  }

  private String collectAnswers(final String... answers) {
    return stream(answers).collect(joining(System.lineSeparator()));
  }
}

With JUnit Pioneer, we don't need to worry about hijacking the streams:

class Let_Junit_Pioneer_Do_Its_Thing_Test {
  @ParameterizedTest
  @ValueSource(strings = {"yes", "sure"})
  @StdIo
  void verify_replying_with_an_acceptable_answer(final String answer, final StdOut output) {
    var origIn = System.in;

    System.setIn(inputStreamOf(answer));
    App.main(new String[0]);

    assertArrayEquals(
      new String[] {"Hello, do you love Java?", "Me too!"},
      output.capturedLines());

    System.setIn(origIn);
  }

  @Test
  @StdIo({"no", "yes"})
  void verify_replying_with_a_wrong_and_an_acceptable_answer(final StdOut output) {
    App.main(new String[0]);

    assertArrayEquals(
      new String[] {
        "Hello, do you love Java?",
        """
            Hmmm... I'm not sure I understand.
            Please reply with 'yes' or 'sure'.
            Do you love Java?""",
        "Me too!"},
      output.capturedLines());
  }

  private InputStream inputStreamOf(final String input) {
    return new ByteArrayInputStream(input.getBytes(UTF_8));
  }
}

For the second test method, verify_replying_with_a_wrong_and_an_acceptable_answer, we're using the StdIo annotation to hijack the in stream and specify the value to be captured by the stream to two lines, "no" and then "yes."
We also use the arguments resolver to inject our method with a hijacked out stream, which we can use to validate the result.

For the first test method, verify_replying_with_an_acceptable_answer, we're using a propertyless StdIo annotation. This annotation marks our test case for the arguments resolver, but we don't need it to hijack the in stream.
However, we make it hijack the out stream by instructing the arguments resolver to inject it so we can validate the results, as we did in the second test method.

The reason is that the test container produces multiple test cases. One with "yes" as the answer and one with "sure."
This means that the value specified for the in stream differs for each case, meaning we had to hijack the in stream ourselves for this specific test.

Executing any of the above test classes will output the following:

[INFO] '-- JUnit Jupiter [OK]
[INFO]   +-- Manual Hijack System Streams Test [OK]
[INFO]   | +-- verify replying with a wrong and an acceptable answer [OK]
[INFO]   | '-- verify replying with an acceptable answer (String) [OK]
[INFO]   |   +-- [1] yes [OK]
[INFO]   |   '-- [2] sure [OK]
[INFO]   '-- Let Junit Pioneer Do Its Thing Test [OK]
[INFO]     +-- verify replying with a wrong and an acceptable answer (StdOut) [OK]
[INFO]     '-- verify replying with an acceptable answer (String, StdOut) [OK]
[INFO]       +-- [1] yes [OK]
[INFO]       '-- [2] sure [OK]

Checkout a previous article of mine, Property-Based Matrix Testing in Java and see how we can turn this:

  @ParameterizedTest
  @MethodSource("getArguments")
  void using_junit_parameterized_test_with_method_source(
      final Direction direction, final Status status) {
    assertTrue(true);
  }

  static Stream<Arguments> getArguments() {
    return Stream.of(Direction.values())
        .flatMap(d -> Stream.of(Status.values()).map(s -> arguments(d, s)));
  }

Into this:

  @CartesianProductTest
  @CartesianEnumSource(Direction.class)
  @CartesianEnumSource(Status.class)
  void using_junit_pioneer_cartesian_product_test_with_enum_source(
      final Direction direction, final Status status) {
    assertTrue(true);
  }

Check out the code for this tutorial on Github.

Property-Based Matrix Testing in Java

Tomer Figenblat — Tue, 13 Apr 2021 14:12:20 +0000

I've used a common approach for property-based matrix testing in Java. Using JUnit's ParameterizedTest and a MethodSource annotations to point to a method as an argument supplier that spits out a stream representing a matrix.

It's pretty straightforward, but the more parameter types we have in our matrix, the harder it is to read or write the method supplying them. 😵

Let's take a look.
First, these are the two types for our matrix, these should spit out 6 test cases:

  enum Direction {
    INCOMING,
    OUTGOING
  }

  enum Status {
    SUCCESS,
    FAILURE,
    WAITING
  }

Implementing a matrix from these Enums with JUnit's ParameterizedTest and a MethodSource:

  @ParameterizedTest
  @MethodSource("getArguments")
  void using_junit_parameterized_test_with_method_source(
      final Direction direction, final Status status) {
    assertTrue(true);
  }

  static Stream<Arguments> getArguments() {
    return Stream.of(Direction.values())
        .flatMap(d -> Stream.of(Status.values()).map(s -> arguments(d, s)));
  }

Adding members to the existing Enums will dynamically increase the matrix and, therefore, the number of tests performed; there's no need to modify the test code.

But, adding a third type to the matrix and the getArguments method will start losing its readability.

Lately, I discovered JUnit Pioneer, which is a JUnit 5 Extension Pack. Using its CartesianProductTest and CartesianEnumSource annotations, we can implement the same matrix simply and elegantly: 😃

  @CartesianProductTest
  @CartesianEnumSource(Direction.class)
  @CartesianEnumSource(Status.class)
  void using_junit_pioneer_cartesian_product_test_with_enum_source(
      final Direction direction, final Status status) {
    assertTrue(true);
  }

This will spit out the same matrix, only now, adding a third element is quite simple: add another CartesianEnumSource annotation.

You can find other types of sources beside Enums, in JUnit Pioneer's Documentation.

As demonstrated in this repository, executing both matrix tests will print out:

[INFO] '-- JUnit Jupiter [OK]
[INFO]   '-- Property Based Matrix Test [OK]
[INFO]     +-- using junit pioneer cartesian product test with enum source (Direction, Status) [OK]
[INFO]     | +-- [1] INCOMING, SUCCESS [OK]
[INFO]     | +-- [2] INCOMING, FAILURE [OK]
[INFO]     | +-- [3] INCOMING, WAITING [OK]
[INFO]     | +-- [4] OUTGOING, SUCCESS [OK]
[INFO]     | +-- [5] OUTGOING, FAILURE [OK]
[INFO]     | '-- [6] OUTGOING, WAITING [OK]
[INFO]     '-- using junit parameterized test with method source (Direction, Status) [OK]
[INFO]       +-- [1] INCOMING, SUCCESS [OK]
[INFO]       +-- [2] INCOMING, FAILURE [OK]
[INFO]       +-- [3] INCOMING, WAITING [OK]
[INFO]       +-- [4] OUTGOING, SUCCESS [OK]
[INFO]       +-- [5] OUTGOING, FAILURE [OK]
[INFO]       '-- [6] OUTGOING, WAITING [OK]

Check out the code for this tutorial in Github.

Alexa Skills Testing with Java

Tomer Figenblat — Sun, 21 Feb 2021 13:13:44 +0000

Test your Java Alexa skills

This tutorial will help you write automated test cases for Alexa skills written in Java. Using real-life JSON requests and verifying multi-turn interactions without deploying or hosting the skill.

To use the alexa-skills-tester, just add the following to your project:
(replace VERSION with the current release).

<dependency>
  <groupId>info.tomfi.alexa</groupId>
  <artifactId>alexa-skills-tester</artifactId>
  <version>VERSION</version>
</dependency>

Example usage

Using a basic Nice to meet you custom skill:
When launching the skill, it asks for the user name and then replies with a Nice to meet you #name prompt, ending the session.

Let's take a look at the code for the skill.

A launch request handler

public final class LaunchRequestHandlerImpl implements LaunchRequestHandler {
  @Override
  public boolean canHandle(final HandlerInput input, final LaunchRequest request) {
    return true;
  }

  @Override
  public Optional<Response> handle(final HandlerInput input, final LaunchRequest request) {
    return input.getResponseBuilder()
        .withSpeech("What is your name?")
        .withReprompt("Please tell me your name.")
        .withShouldEndSession(false)
        .build();
  }
}

An intent request handler handling `MyNameIntent` with a `nameSlot`

public final class MyNameIntentRequestHandler implements IntentRequestHandler {
  @Override
  public boolean canHandle(final HandlerInput input, final IntentRequest request) {
    return request.getIntent().getName().equals("MyNameIntent");
  }

  @Override
  public Optional<Response> handle(final HandlerInput input, final IntentRequest request) {
    var name = request.getIntent().getSlots().get("nameSlot").getValue();
    return input.getResponseBuilder()
        .withSpeech(String.format("Nice to meet you %s!", name))
        .withShouldEndSession(true)
        .build();
  }
}

A session ended request handler

public final class SessionEndedRequestHandlerImpl implements SessionEndedRequestHandler {
  @Override
  public boolean canHandle(final HandlerInput input, final SessionEndedRequest request) {
    return true;
  }

  @Override
  public Optional<Response> handle(final HandlerInput input, final SessionEndedRequest request) {
    return Optional.empty();
  }
}

A utility class for constructing the skill

class NiceToMeetYouSkill {
  public static Skill getSkill() {
    return Skills.standard()
        .addRequestHandler(new LaunchRequestHandlerImpl())
        .addRequestHandler(new MyNameIntentRequestHandler())
        .addRequestHandler(new SessionEndedRequestHandlerImpl())
        .build();
  }
}

Now, let's test the skill

You should get familiarized with Alexa's Request and Response JSON Reference. This is, of course, should be considered common knowledge for skill developers.

Let's create a couple of fake request JSON files to use in our tests.

launch_request.json

{
  "version": "1.0",
  "session": {
    "new": true
  },
  "context": {
    "System": {}
  },
  "request": {
    "type": "LaunchRequest",
    "requestId": "amzn1.echo-api.request.fake-request-id",
    "timestamp": "2021-02-11T15:30:00Z",
    "locale": "en-US"
  }
}

my_name_intent.json

{
  "version": "1.0",
  "session": {
    "new": false
  },
  "context": {
    "System": {}
  },
  "request": {
    "type": "IntentRequest",
    "requestId": "amzn1.echo-api.request.fake-request-id",
    "timestamp": "2021-02-11T15:31:00Z",
    "locale": "en-US",
    "intent": {
      "name": "MyNameIntent",
      "slots": {
        "nameSlot": {
          "name": "nameSlot",
          "value": "master"
        }
      }
    }
  }
}

session_ended.json

{
    "version": "1.0",
    "session": {
        "new": false
    },
    "context": {
        "System": {}
    },
    "request": {
        "type": "SessionEndedRequest",
        "requestId": "amzn1.echo-api.request.fake-request-id",
        "timestamp": "2021-02-11T15:32:00Z",
        "locale": "en-US"
    }
}

Now, let's create some test cases leveraging the above JSON files to verify the interaction with the skill.

Test cases testing the `Nice to meet you` skill

final class SkillInteractionTest {
  @Test
  void verify_greeting_for_my_name_intent_as_a_followup_to_a_launch_request() throws IOException {
    givenSkill(NiceToMeetYouSkill.getSkill())
        .whenRequestIs(getClass().getClassLoader().getResourceAsStream("launch_request.json").readAllBytes())
        .thenResponseShould()
            .waitForFollowup()
            .haveOutputSpeechOf("What is your name?")
            .haveRepromptSpeechOf("Please tell me your name.")
        .followingUpWith(getClass().getClassLoader().getResourceAsStream("my_name_intent.json").readAllBytes())
        .thenResponseShould()
            .haveOutputSpeechOf("Nice to meet you master!")
            .and()
            .notWaitForFollowup();
  }

  @Test
  void verify_empty_response_for_session_ended_requests() throws IOException {
    givenSkill(NiceToMeetYouSkill.getSkill())
      .whenRequestIs(getClass().getClassLoader().getResourceAsStream("launch_request.json").readAllBytes())
      .thenResponseShould()
          .waitForFollowup()
          .haveOutputSpeechOf("What is your name?")
          .haveRepromptSpeechOf("Please tell me your name.")
      .followingUpWith(getClass().getClassLoader().getResourceAsStream("session_ended.json").readAllBytes())
      .thenResponseShould().beEmpty();
  }
}

That's it!
I hope you'll find this tool useful.

Migrating to Java Platform Module System

Tomer Figenblat — Tue, 12 Jan 2021 22:38:22 +0000

JPMS Migration Playground

This tutorial presents a complex scenario of migrating a non-modular project to JPMS. Inspired by a scenario I previously faced where one of my transitive dependencies was in-accessible (and not required).

Scenario Walkthrough

Project bar is the missing artifact used by foo. It is configured to be deployed to foo's lib folder as a non-modular jar.

Project foo is the artifact we require in our project, baz. It is configured to be deployed to baz's lib folder as a non-modular jar.

To work around the obvious compliation error of trying to compile foo when bar is inaccessible, we used the flatten plugin to strip foo's pom from its dependencies so that bar will not be known at compile time to whoever uses foo.

Basically, baz depends on foo, without depending on bar and the code compiles successfully. 😁
It will, however, throw a NoClassDefFoundError exception if we were to try an access bar's classes. This is demonstrated in BazTest.java.

Pre-JPMS code is in this commit.

JPMS: foo as Automatic Module

We can get foo in our modulepath, implicitly making it an automatic module by explicitly requiring it by its unstable name. 😵
Demonstrated in this commit.

Note that when compiling, the compiler will inform us of the automatic module usage:

[INFO] Required filename-based automodules detected: [foo-0.0.1.jar]. Please don't publish this project to a public artifact repository!

This means our module won't be a pure module. And we won't be able to make the best out of JPMS. Features like jlink don't work with automatic modules. 😞

Migrating foo

We use the moditect plugin to tweak foo. However, in our case, we can't access bar. Although our project, baz, doesn't require it, it's required by foo.

For clarification:

Our project in the works is baz.
The non-modular jar foo is given to us (without the source files).
The non-modular jar bar is not given to us, nor do we require it.

We aim to make foo modular while ignoring bar so that baz can truly leverage JPMS features while requiring foo.

We're using the junit-platform plugin for testing.

Create a module-info descriptor

The first step is to create a module-info descriptor for foo. This can be achieved in two ways.

From the command line (note the --ignore-missing-deps because we don't have bar):

jdeps --generate-module-info .\target\descs --ignore-missing-deps .\lib\com\example\foo\0.0.1\foo-0.0.1.jar

This will create .\target\descs\foo\module-info.java:

module foo {
    exports com.example.foo;
}

The same can also be achieved in build time using the moditect plugin. Better yet, we can now give the module a stable name.
By convention, let's name it com.example.foo:

<plugin>
    <groupId>org.moditect</groupId>
    <artifactId>moditect-maven-plugin</artifactId>
    <version>1.0.0.RC1</version>
    <executions>
        <execution>
            <id>generate-module-info</id>
            <phase>initialize</phase>
            <goals>
                <goal>generate-module-info</goal>
            </goals>
            <configuration>
                <outputDirectory>${project.build.directory}/descs</outputDirectory>
                <modules>
                    <module>
                        <artifact>
                            <groupId>com.example</groupId>
                            <artifactId>foo</artifactId>
                            <version>0.0.1</version>
                        </artifact>
                        <moduleInfo>
                            <name>com.example.foo</name>
                        </moduleInfo>
                    </module>
                </modules>
                <jdepsExtraArgs>--ignore-missing-deps</jdepsExtraArgs>
            </configuration>
        </execution>
    </executions>
</plugin>

Create a modular jar

Now that we have our module-info descriptor ready, we can create a new modular jar with it.

This is easily achived with the moditect plugin:

<plugin>
    <groupId>org.moditect</groupId>
    <artifactId>moditect-maven-plugin</artifactId>
    <version>1.0.0.RC1</version>
    <executions>
        <execution>
            <id>add-module-info</id>
            <phase>initialize</phase>
            <goals>
                <goal>add-module-info</goal>
            </goals>
            <configuration>
                <modules>
                    <module>
                        <artifact>
                            <groupId>com.example</groupId>
                            <artifactId>foo</artifactId>
                        </artifact>
                        <moduleInfoFile>${project.build.directory}/descs/com.example.foo/module-info.java</moduleInfoFile>
                    </module>
                </modules>
            </configuration>
        </execution>
    </executions>
</plugin>

This will create a modified version foo-0.0.1.jar in target\modules. The modified version will include the following module-info descriptor and will qualify as a named module.

module com.example.foo {
    exports com.example.foo;
}

Use the modular jar

To use our new modular jar, we update the requires statement in the source files of the baz project. From foo, the automatic module unstable name, to com.example.foo, the named module stable name.

// original using an automatic module
module com.example.baz {
  requires foo;
}
// modified using the new named module
module com.example.baz {
  requires com.example.foo;
}

A modification is also required in the test sources descriptor:

// original using an automatic module
open module com.example.baz {
  requires foo;
}
// modified using the new named module
open module com.example.baz {
  requires com.example.foo;

  requires org.junit.jupiter.api;
}

Note that for the test descriptor, we needed to add a requires directive for reading org.junit.jupiter.api.

We didn't need to do so before because as an automatic module, foo could read all the other modules from the modulepath. So it bridged between the modules com.example.baz and org.junit.jupiter.api, so we didn't need to explicitly make com.example.baz read org.junit.jupiter.api.

Now that com.example.foo is a named module, com.example.baz needs to explicitly require org.junit.jupiter.api.

Failing to do so will result in a compilation error:

[ERROR] .../src/test/java/com/example/baz/BazTest.java:[3,32] package org.junit.jupiter.api is not visible

Add the new module to the Modulepath

This is where it gets tricky. Typically, we would configure the compiler to include the new module like so:

<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <compilerArgs>
            <arg>--upgrade-module-path</arg>
            <arg>${project.build.directory}/modules</arg>
        </compilerArgs>
    </configuration>
</plugin>

This, unfortunately, will not work in our case. 😟

The original modulepath is constructed from the classpath, every monolithic jar is treated as part of the unnamed module. This means our jar will exist twice on the modulepath, as part of the unnamed module and as a named module named com.example.foo.

When trying to compile the test classes that access the module, we'll get the obvious error:

[ERROR] .../src/test/java/module-info.java:[1,6] module com.example.baz reads package com.example.foo from both com.example.foo and foo

To avoid the above error, we cannot do upgrade-module-path. We must reconstruct the modulepath from scratch, excluding the original jar.

Create a new Classpath

First, we use the dependency plugin to create a temporary file with the classpath content excluding foo in target\fixedClasspath.txt:

<plugin>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
    <execution>
        <id>create-classpath-file</id>
        <goals>
            <goal>build-classpath</goal>
        </goals>
        <configuration>
            <outputFile>${project.build.directory}/fixedClasspath.txt</outputFile>
            <excludeArtifactIds>foo</excludeArtifactIds><!-- every aritifact recreated with moditect should be listed here -->
        </configuration>
    </execution>
    </executions>
</plugin>

Create a new Modulepath

We then use the gmavenplus plugin to execute a small Groovy script creating a property containing the fixed modulepath.

We use the os plugin to create the os.detected.name property.

<extension>
    <groupId>kr.motd.maven</groupId>
    <artifactId>os-maven-plugin</artifactId>
    <version>1.6.2</version>
</extension>

And execute a Groovy script creating a property named modulePath containing the content of the fixed classpath and the named module we previously created:

<plugin>
    <groupId>org.codehaus.gmavenplus</groupId>
    <artifactId>gmavenplus-plugin</artifactId>
    <version>1.12.0</version>
    <dependencies>
        <dependency>
            <groupId>org.codehaus.groovy</groupId>
            <artifactId>groovy-ant</artifactId>
            <version>3.0.7</version>
            <scope>runtime</scope>
        </dependency>
    </dependencies>
    <executions>
        <execution>
        <phase>process-sources</phase>
        <goals>
            <goal>execute</goal>
        </goals>
        <configuration>
            <scripts>
                <script><![CDATA[
                    def delimiter = project.properties['os.detected.name'] == 'windows' ? ';' : ':'
                    def file = new File("$project.build.directory/fixedClasspath.txt")
                    project.properties.setProperty 'modulePath', file.text + delimiter + "$project.build.directory/modules"
                ]]></script>
            </scripts>
        </configuration>
        </execution>
    </executions>
</plugin>

Patch compiler with new Modulepah

As simple as:

<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <compilerArgs>
            <arg>--module-path</arg>
            <arg>${modulePath}</arg>
        </compilerArgs>
    </configuration>
</plugin>

Patch testing Classpath

Next, we must exclude the original monolithic jar foo and include the modular one on the test classpath. This can be achieved with junit-platform plugin tweaks:

<plugin>
    <groupId>de.sormuras.junit</groupId>
    <artifactId>junit-platform-maven-plugin</artifactId>
    <version>1.1.0</version>
    <extensions>true</extensions>
    <configuration>
        <executor>JAVA</executor>
        <tweaks>
            <additionalTestPathElements>
                <element>${project.build.directory}/modules/foo-0.0.1.jar</element>
            </additionalTestPathElements>
            <dependencyExcludes>
                <exclude>com.example:foo</exclude>
            </dependencyExcludes>
        </tweaks>
    </configuration>
</plugin>

That's it.
Everything now compiles, and all the tests pass.
😀

The key added value here is that both the project baz and the local dependency foo are now named modules, although we are still missing bar, of course.
We can also use moditect to make foo stop exposing packages related to bar so we won't be able to access them by mistake.
😎

I had fun playing around with JPMS; I hope you did, too.

You can check out the code for this playground in Github.

My Ultimate Development Container image

Tomer Figenblat — Fri, 14 Aug 2020 11:19:43 +0000

My Ultimate Development Container Image

Using VSCode's Remote Development Feature, we can remotely develop on another station using SSH or WSL. But my favorite feature is using Containers.
We can potentially create isolated development environments per project in a container.

I've cultivated a Container image I use throughout my projects. It includes my most frequently used binaries and tools.

The key here is the layers; they were pre-designed in a way so they could be sharable for all of my containers. Basically:

Environment variables go at the lowest layer as they rarely get modified.
Binaries installations go in the next layers separately from one another, so when modified, they'll require only the layers above them to be rebuilt.
Modules, requirements, dependencies, packages, libraries, etc., go in the top layer as they get modified more frequently.

You can check out the Dockerfile source gist.
😎

# Base image is Python's image
FROM python:3.8.5

# Set shell with pipeline fail return
SHELL ["/bin/bash", "-o", "pipefail", "-c"]

# Set environment variables
ENV SHELL=/bin/bash \
    DOTNET_CLI_TELEMETRY_OPTOUT=true \
    JAVA_HOME=/opt/java \
    MAVEN_HOME=/opt/maven \
    PATH_TO_FX=/usr/share/java/ \
    PATH=$PATH:/opt/java/bin:/opt/maven/bin:/opt/gopath/bin:/opt/go/bin:/root/.dotnet/tools

# Install Linux packages (including dotnet and javafx)
RUN curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.asc.gpg \
    && mv microsoft.asc.gpg /etc/apt/trusted.gpg.d/ \
    && curl -sL https://packages.microsoft.com/config/debian/10/prod.list \
        -o /etc/apt/sources.list.d/microsoft-prod.list \
    && apt-get update \
    && apt-get install -y --no-install-recommends apt-transport-https \
    && apt-get install -y --no-install-recommends dotnet-sdk-3.1 \
    && apt-get install -y --no-install-recommends git-lfs \
    && apt-get install -y --no-install-recommends nano \
    && apt-get install -y --no-install-recommends openjfx \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Install java
WORKDIR /opt/java
RUN curl -sOL https://github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.7%2B10_openj9-0.20.0/OpenJDK11U-jdk_x64_linux_openj9_11.0.7_10_openj9-0.20.0.tar.gz \
    && tar -xf OpenJDK11U-jdk_x64_linux_openj9_11.0.7_10_openj9-0.20.0.tar.gz \
    && rm OpenJDK11U-jdk_x64_linux_openj9_11.0.7_10_openj9-0.20.0.tar.gz \
    && mv jdk-11.0.7+10/* . \
    && rm -r jdk-11.0.7+10

# Install maven
WORKDIR /opt/maven
RUN curl -sOL https://downloads.apache.org/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz \
    && tar -xf apache-maven-3.6.3-bin.tar.gz \
    && rm apache-maven-3.6.3-bin.tar.gz \
    && mv apache-maven-3.6.3/* . \
    && rm -r apache-maven-3.6.3

# Install Node.js (includes npm)
RUN curl -sL https://deb.nodesource.com/setup_14.x | bash - \
    && apt-get install -y --no-install-recommends nodejs \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*

# Install golang
WORKDIR /opt/go
RUN curl -sOL https://dl.google.com/go/go1.14.4.linux-amd64.tar.gz \
    && tar -C /opt -xf go1.14.4.linux-amd64.tar.gz \
    && rm go1.14.4.linux-amd64.tar.gz \
    && bin/go env -w GOPATH=/opt/gopath GO111MODULE=on

# Install modules, requirements, dependencies, packages, libraries, etc.
RUN \
    # Install node modules
    npm install -g \
        commitizen@4.1.2 \
        cz-conventional-changelog@3.2.0 \
        @commitlint/cli@9.1.1 \
        @commitlint/config-conventional@9.1.1 \
        dockerfilelint@1.8.0 \
        jsonlint@1.6.3 \
        markdownlint-cli@0.23.2 \
        semantic-release@17.1.1 \
        yarn@1.22.4 \
    # Install go modules
    && go get -u \
        github.com/bufbuild/buf/cmd/buf@v0.20.5 \
        github.com/bufbuild/buf/cmd/protoc-gen-buf-check-breaking@v0.20.5 \
        github.com/bufbuild/buf/cmd/protoc-gen-buf-check-lint@v0.20.5 \
        github.com/client9/misspell/cmd/misspell@v0.3.4 \
        github.com/pseudomuto/protoc-gen-doc/cmd/protoc-gen-doc@v1.3.2 \
        github.com/yoheimuta/protolint/cmd/protolint@v0.26.0 \
    # Install python dependencies
    && pip install \
        docutils==0.16 \
        pre-commit==2.6.0 \
        robotframework==3.2.1 \
        yamllint==1.24.2 \
    # Install dotnet tools
    && dotnet tool install -g dotnet-format --version 4.1.131201 \
    && dotnet tool install -g dotnet-reportgenerator-globaltool --version 4.6.4

I hope you find this helpful.

Dependency Injection in Java is easy - Part 3 - Spring Context

Tomer Figenblat — Fri, 14 Aug 2020 07:36:55 +0000

Part 3 - Spring Context

This post is part of a multiple-part tutorial. As the heading suggests, this part will focus on Dependency Injection using Spring Context.

Note, Spring is so much more than a DI Framework, but this tutorial revolves around Spring Framework IoC Container, also known as Spring Context.
Spring Boot or any other Spring component is outside the scope of this tutorial.

You can check out the code for this tutorial part in Github.

Background

It's advised to start with Part 1 - A Design Pattern, which this part requires. You can skip Part 2 - Google Guice.

If you have read Part 2, you can skip to the incorporating spring section.

Dependency Injection Frameworks

DI Frameworks concept is pretty straightforward; the framework creates and manages our dependencies for us.
On one end, we provide the framework instructions for creating our dependencies; on the other, we ask the framework for instances of the dependencies.

Comparing multiple frameworks, we'll notice different features, component names, default behaviors, and probably different implementations under the hood. But the gist will be the same: a container that holds our dependencies. Let's explore some similarities between various frameworks:

Spring's Autowired and Guice's Inject, both mark a dependency for injection.
Spring's Bean and Guice's Provides, both incorporate logic for creating a dependency.

Not a believer yet ❔ Alright, let's throw C# in the mix ❕

Spring's Configuration, Guice's Modules, and Autofac's Modules, can be used to configure dependencies.
Guice's Binding and Autofac's Registration are used to configure dependencies with the framework's DSL.

We can go on. But the point is made.
😎

Under the hood, DI Frameworks builds factories providing dependencies based on multiple criteria, such as type and name.

At the base level, there are three typical scopes for dependencies living in a DI Framework container:

Eager Singleton: one instance of the dependency will be created immediately upon the framework's instantiation; the same instance will be provided for every request.
Lazy Singleton: one instance of the dependency will be created only when requested. Upon its instantiation, the same instance will be provided for every request.
Non-Singleton: A new instance of the dependency will be provided for every request.

There are more scopes, but these are the three most commonly used. Some frameworks offer different scopes than others. Such as per session, per request, per dependency, etc.

Example App

Heads up: The example app next is based on Part 1 - A Design Pattern; you can skip to the incorporating spring section.

Mail Collector App

Let's build an app pulling emails from both Gmail and Microsoft leveraging the Dependency Injection pattern with Spring Context.

Contracts

An Enum called MailSource for categorizing the email source:

public enum MailSource {
  GMAIL,
  MICROSOFT;
}

An abstract class Mail for contracting mail objects.

public abstract class Mail {
  public abstract String from();

  public abstract String subject();

  public abstract MailSource source();

  @Override
  public String toString() {
    return String.format("Got mail by %s, from %s, with the subject %s", source(), from(), subject());
  }
}

An interface for contracting services responsible for pulling Mail from suppliers, the MailService.

public interface MailService {
  List<Mail> getMail();
}

And last, an interface for contracting an engine responsible for collecting Mail from multiple services, the MailEngine.

public interface MailEngine {
  List<Mail> getAllMail();
}

Implementations

Mail

The concrete Mail implementations were designed with a builder pattern for convenience and immutability.
The Gmail Mail implementation, GmailImpl:

public final class GmailImpl extends Mail {
  private final String setFrom;
  private final String setSubject;

  private GmailImpl(final String from, final String subject) {
    setFrom = from;
    setSubject = subject;
  }

  @Override
  public String from() {
    return setFrom;
  }

  @Override
  public String subject() {
    return setSubject;
  }

  @Override
  public MailSource source() {
    return MailSource.GMAIL;
  }

  public static GmailImpl.Builder builder() {
    return new GmailImpl.Builder();
  }

  public static final class Builder {
    private String prepFrom;
    private String prepSubject;

    public Builder from(final String setFrom) {
      prepFrom = setFrom;
      return this;
    }

    public Builder subject(final String setSubject) {
      prepSubject = setSubject;
      return this;
    }

    public GmailImpl build() {
      requireNonNull(emptyToNull(prepFrom), "from cannot be empty or null");
      requireNonNull(emptyToNull(prepSubject), "subject cannot be empty or null");

      return new GmailImpl(prepFrom, prepSubject);
    }
  }
}

The Micsorosft Mail implementation, MicrosoftImpl:

public final class MicrosoftImpl extends Mail {
  private final String setFrom;
  private final String setSubject;

  private MicrosoftImpl(final String from, final String subject) {
    setFrom = from;
    setSubject = subject;
  }

  @Override
  public String from() {
    return setFrom;
  }

  @Override
  public String subject() {
    return setSubject;
  }

  @Override
  public MailSource source() {
    return MailSource.MICROSOFT;
  }

  public static MicrosoftImpl.Builder builder() {
    return new MicrosoftImpl.Builder();
  }

  public static final class Builder {
    private String prepFrom;
    private String prepSubject;

    public Builder from(final String setFrom) {
      prepFrom = setFrom;
      return this;
    }

    public Builder subject(final String setSubject) {
      prepSubject = setSubject;
      return this;
    }

    public MicrosoftImpl build() {
      requireNonNull(emptyToNull(prepFrom), "from cannot be empty or null");
      requireNonNull(emptyToNull(prepSubject), "subject cannot be empty or null");

      return new MicrosoftImpl(prepFrom, prepSubject);
    }
  }
}

Mail Services

The Gmail MailService implementation:

public final class GmailService implements MailService {
  @Override
  public List<Mail> getMail() {
    //This is where the actual Gmail API access goes.
    //We'll fake a couple of emails instead.
    var firstFakeMail =
        GmailImpl.builder()
            .from("a.cool.friend@gmail.com")
            .subject("wanna get together and write some code?")
            .build();

    var secondFakeMail =
        GmailImpl.builder()
            .from("an.annoying.salesman@some.company.com")
            .subject("wanna buy some stuff?")
            .build();

    return List.of(firstFakeMail, secondFakeMail);
  }
}

The Microsoft MailService implementation:

public final class MicrosoftService implements MailService {
  @Override
  public List<Mail> getMail() {
    //This is where the actual Microsoft API access goes.
    //We'll fake a couple of emails instead.
    var firstFakeMail =
        MicrosoftImpl.builder()
            .from("my.boss@work.info")
            .subject("stop writing tutorials and get back to work!")
            .build();

    var secondFakeMail =
        MicrosoftImpl.builder()
            .from("next.door.neighbor@kibutz.org")
            .subject("do you have philips screwdriver?")
            .build();

    return List.of(firstFakeMail, secondFakeMail);
  }
}

Mail Engine

public final class RobustMailEngine implements MailEngine {
  private final Set<MailService> mailServices;

  public RobustMailEngine(final Set<MailService> setMailSerices) {
    mailServices = setMailSerices;
  }

  @Override
  public List<Mail> getAllMail() {
    return mailServices.stream().map(MailService::getMail).flatMap(List::stream).collect(toList());
  }
}

The Main App

This is the app itself, the MailCollectorApp:

public final class MailCollectorApp {
  private MailEngine engine;

  public MailCollectorApp(final MailEngine setEngine) {
    engine = setEngine;
  }

  public String getMail() {
    var ret = "No mail found.";
    if (!engine.getAllMail().isEmpty()) {
      ret = Joiner.on(System.lineSeparator()).join(engine.getAllMail());
    }
    return ret;
  }

  public static void main(final String... args) {
    var gmailService = new GmailService();
    var microsoftService = new MicrosoftService();

    var engine = new RobustMailEngine(Set.of(gmailService, microsoftService));

    var app = new MailCollectorApp(engine);

    System.out.println(app.getMail());
  }
}

Executing the main method will print:

Got Mail by GMAIL, from a.cool.friend@gmail.com, with the subject wanna get together and write some code?
Got Mail by GMAIL, from an.annoying.salesman@some.company.com, with the subject wanna buy some stuff?
Got Mail by MICROSOFT, from my.boss@work.info, with the subject stop writing tutorials and get back to work!
Got Mail by MICROSOFT, from next.door.neighbor@kibutz.org, with the subject do you have a star screwdriver?

This application uses the dependency injection design pattern. The dependencies are currently controlled by the main method, so it should be easy to incorporate Spring Context.

Incorporating Spring Context

Include maven dependency

First, let's add this to our pom.xml in the dependencies section:
Note that this version was the latest when this tutorial was written.

<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-context</artifactId>
  <version>5.2.8.RELEASE</version>
</dependency>

Mark Autowired

We need to tell Spring about the dependencies we want to be injected. But it's only mandatory for property and method based injections; in our example, we're using constructor based injection, so there's no need to annotate with @Autowierd.

But for this tutorial's sake, we'll use the @Autowierd to demonstrate what dependencies we want to be injected. Let's do this for the concrete engine class:

public final class RobustMailEngine implements MailEngine {
  private final Set<MailService> mailServices;

  @Autowired
  public RobustMailEngine(final Set<MailService> setMailSerices) {
    mailServices = setMailSerices;
  }

  @Override
  public List<Mail> getAllMail() {
    return mailServices.stream().map(MailService::getMail).flatMap(List::stream).collect(toList());
  }
}

And for the app class:

public final class MailCollectorApp {
  private MailEngine engine;

  @Autowired
  public MailCollectorApp(final MailEngine setEngine) {
    engine = setEngine;
  }

  public String getMail() {
    var ret = "No mail found.";
    if (!engine.getAllMail().isEmpty()) {
      ret = Joiner.on(System.lineSeparator()).join(engine.getAllMail());
    }
    return ret;
  }

  //...
}

Now, we need to instruct Spring on instantiating those dependencies, called beans in spring's world.

Create Beans

Let's create a Spring Configuration Class for creating beans; multiple approaches exist to achieve that. For instance, using Spring's Annotation Based Configuration:

@Configuration
public class DIConfiguration {
  @Bean
  @Scope(BeanDefinition.SCOPE_PROTOTYPE)
  public Set<MailService> getServices() {
    return Set.of(new GmailService(), new MicrosoftService());
  }

  @Lazy
  @Bean
  public MailEngine getEngine(final Set<MailService> services) {
    return new RobustMailEngine(services);
  }
}

Another approach to creating beans in Spring is marking classes as @Component, which is how to tell Spring we want this class as a bean. For instance, let's mark our app class as a Component so we can later ask Spring to instantiate it:

@Lazy
@Component
public final class MailCollectorApp {
  // ...
}

This will, of course, be a Lazy Singleton.

Another approach to configuring Spring is using Spring's XML Based Configuration:

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="
        http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

    <bean id="gmailService" class="info.tomfi.tutorials.mailapp.core.service.GmailService" scope="prototype"/>

    <bean id="microsoftService" class="info.tomfi.tutorials.mailapp.core.service.MicrosoftService" scope="prototype"/>

    <bean id="getEngine" class="info.tomfi.tutorials.mailapp.engine.RobustMailEngine" lazy-init="true">
      <constructor-arg>
        <set>
          <ref bean="gmailService"/>
          <ref bean="microsoftService"/>
        </set>
      </constructor-arg>
    </bean>

    <bean id="getMailApp" class="info.tomfi.tutorials.mailapp.MailCollectorApp" lazy-init="true">
      <constructor-arg>
        <ref bean="getEngine"/>
      </constructor-arg>
    </bean>

</beans>

Both options will produce the same dependencies.

Spring's default scope is Singleton. This means that the RobustMailEngine, combined with the Lazy annotation, will be a Lazy Singleton, meaning our app will have only one instance of RobustMailEngine.

On the other end, the Set of GmailService and MicrosoftService will be created as new instances for every object that needs them, as we explicitly configured them as Prototype, which is Non-Singleton.

Note, RobustMailEngine is a dependency; it will be injected to whoever requests it, but as we configured earlier, it also needs dependencies for itself (the Set of MailService).

Spring will catch that and inject the set of mail services while instantiating the engine.

Update the app to use Spring

Getting back to our app. Let's update it to work with Spring:

@Lazy
@Component
public final class MailCollectorApp {
  private MailEngine engine;

  public MailCollectorApp(final MailEngine setEngine) {
    engine = setEngine;
  }

  public String getMail() {
    var ret = "No mail found.";
    if (!engine.getAllMail().isEmpty()) {
      ret = Joiner.on(System.lineSeparator()).join(engine.getAllMail());
    }
    return ret;
  }

  public static void main(final String... args) {
    // try (var container = new ClassPathXmlApplicationContext("spring-beans.xml")) {
    try (var container = new AnnotationConfigApplicationContext(MailCollectorApp.class, DIConfiguration.class)) {
      var app = container.getBean(MailCollectorApp.class);

      System.out.println(app.getMail());
    }
  }
}

Let's analyze what's going on here...

Running the main method will create a Spring container, the context for all dependencies. As we create it with an instance of both MailCollectorApp and DIConfiguration classes, the container will have the following dependencies, beans, configured in it:

A Set of two MailService objects (GmailService and Microsoft Service).
A Singleton instance of MailEngine (RobustMailEngine).
A Singleton instance of MailCollectorApp.

As configured, both MailCollectorApp and MainEngine are Lazy Singletons, plus, we configured the Set of MailService as Prototype. That means that there's nothing instantiated in Spring's context at this point.

The next step, asking the container for an instance of MailCollectorApp, will accomplish the following:

Spring will pick up the constructor in MailCollectorApp, as it's the only constructor.
Spring will look in its context for a type MailEngine dependency.
It will find the RobustMailEngine configured, which is a Lazy Singleton.
While trying to instantiate it, it will pick up its constructor and look for a suitable dependency with the type Set of MailService.
It will find the Set of GoogleService and MicrosoftService, which are Prototypes.

After preparing the groundwork, Spring will:

Create the set after instantiating both GmailService and MicrosoftService.
Instantiate the RobustMailEngine injecting the Set.
Instantiate the MailCollectorApp injecting the RobustMailEngine.

We then get our instance MailCollectorApp with everything we need, from which we invoke getMail to get all our Mail.

That's it, Spring Context in a nutshell.
😆

Now, Let's test our code.

Unit Tests

I will start by saying that when it comes to unit tests, if possible, I always prefer not to use DI Frameworks.

Unit tests are about testing small parts, units, of our application. We can avoid the overhead of creating the DI Context. We'll be better off simply instantiating the subject under test manually.

On the other end, if we're writing integration, acceptance tests, or any different situation when we might need to test our application end-to-end, well, in that case, a suitable DI Framework could be our best friend.

Let's move on to unit tests with a DI Framework for demonstration purposes only.

Please note that Spring creates its instances and doesn't allow outside interference. Nonetheless, we needed to inject mocks instead of the real mail services, maintaining the ability to access them and assert their behavior. We create a separate configuration class for testing Spring Context, having it instantiate the mocks for us:

@Configuration
public class DITestConfiguration {
  private MailService gmailServiceMock;
  private MailService microsoftServiceMock;
  private MailService thirdServiceMock;

  public DITestConfiguration() {
    gmailServiceMock = mock(MailService.class);
    microsoftServiceMock = mock(MailService.class);
    thirdServiceMock = mock(MailService.class);
  }

  @Lazy
  @Bean
  public MailEngine getEngine(final Set<MailService> services) {
    return new RobustMailEngine(Set.of(gmailServiceMock, microsoftServiceMock, thirdServiceMock));
  }

  public MailService getGmailServiceMock() {
    return gmailServiceMock;
  }

  public MailService getMicrosoftServiceMock() {
    return microsoftServiceMock;
  }

  public MailService getThirdServiceMock() {
    return thirdServiceMock;
  }
}

We then use this specific testing configuration for creating our context:

public final class MailCollectorAppTest {
  private MailService gmailServiceMock;
  private MailService microsoftServiceMock;
  private MailService thirdServiceMock;

  private MailCollectorApp sut;

  private ConfigurableApplicationContext context;
  private Faker faker;

  @BeforeEach
  public void initialize() {
    faker = new Faker();

    context =
        new AnnotationConfigApplicationContext(MailCollectorApp.class, DITestConfiguration.class);

    var confWorkAround = context.getBean(DITestConfiguration.class);

    gmailServiceMock = confWorkAround.getGmailServiceMock();
    microsoftServiceMock = confWorkAround.getMicrosoftServiceMock();
    thirdServiceMock = confWorkAround.getThirdServiceMock();

    sut = context.getBean(MailCollectorApp.class);
  }

  @AfterEach
  public void cleanup() {
    context.close();
  }

  @Test
  @DisplayName(
      "make the services mocks return no mail and validate the return string as 'No mail found'")
  public void getMail_noMailExists_returnsNoMailFound() {
    willReturn(emptyList()).given(gmailServiceMock).getMail();
    willReturn(emptyList()).given(microsoftServiceMock).getMail();
    willReturn(emptyList()).given(thirdServiceMock).getMail();

    then(sut.getMail()).isEqualTo("No mail found.");
  }

  @Test
  @DisplayName(
      "make the services return legitimate mail and validate the return string as expected")
  public void getMail_foundMail_returnsExpectedString() {
    var mail1 =
        GmailImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();
    var mail2 =
        MicrosoftImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();
    var mail3 =
        MicrosoftImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();

    willReturn(List.of(mail1)).given(gmailServiceMock).getMail();
    willReturn(List.of(mail2, mail3)).given(microsoftServiceMock).getMail();
    willReturn(emptyList()).given(thirdServiceMock).getMail();

    then(sut.getMail().split(System.lineSeparator()))
        .containsOnly(mail1.toString(), mail2.toString(), mail3.toString());
  }
}

It's important to emphasize this is poor practice; We were better off testing without Spring, but this tutorial aims to show how to do so for rare cases where there's no better way.

In the code above, we created the container with a DITestConfiguration instance instead of a DIConfiguration instance, exposing three getters to help work around the mock injection issue. We also added a third mail service, demonstrating how easy it is. 😁

The RobustMailEngine was not mocked because there was no real reason to do so, but it could have been easily replaced with a mock or a spy.

The test class behaved exactly like the main app, except for the services being mocks instead of real objects.

Dependency Injection in Java is easy - Part 2 - Google Guice

Tomer Figenblat — Sun, 09 Aug 2020 21:13:29 +0000

Part 2 - Google Guice

This post is part of a multiple-part tutorial. As the heading suggests, this part will focus on Dependency Injection using Google Guice.

You can check out the code for this tutorial part in Github.

Background

It's advised to start with Part 1 - A Design Pattern, which this part requires. The next Part 3 - Spring Context doesn't require this one.

If you have read [Part 2][2], you can skip to the incorporating guice section.

Dependency Injection Frameworks

Spring's Autowired and Guice's Inject, both mark a dependency for injection.
Spring's Bean and Guice's Provides, both incorporate logic for creating a dependency.

Not a believer yet ❔ Alright, let's throw C# in the mix ❕

Spring's Configuration, Guice's Modules, and Autofac's Modules, can be used to configure dependencies.
Guice's Binding and Autofac's Registration are used to configure dependencies with the framework's DSL.

We can go on. But the point is made.
😎

Under the hood, DI Frameworks builds factories providing dependencies based on multiple criteria, such as type and name.

At the base level, there are three typical scopes for dependencies living in a DI Framework container:

Eager Singleton: one instance of the dependency will be created immediately upon the framework's instantiation; the same instance will be provided for every request.
Lazy Singleton: one instance of the dependency will be created only when requested. Upon its instantiation, the same instance will be provided for every request.
Non-Singleton: A new instance of the dependency will be provided for every request.

There are more scopes, but these are the three most commonly used. Some frameworks offer different scopes than others. Such as per session, per request, per dependency, etc.

Example App

Heads up: The example app next is based on Part 1 - A Design Pattern; you can skip to the incorporating guice section.

Mail Collector App

Let's build an app pulling emails from both Gmail and Microsoft leveraging the Dependency Injection pattern with Google Guice.

Contracts

An Enum called MailSource for categorizing the email source:

public enum MailSource {
  GMAIL,
  MICROSOFT;
}

An abstract class Mail for contracting mail objects.

public abstract class Mail {
  public abstract String from();

  public abstract String subject();

  public abstract MailSource source();

  @Override
  public String toString() {
    return String.format("Got mail by %s, from %s, with the subject %s", source(), from(), subject());
  }
}

An interface for contracting services responsible for pulling Mail from suppliers, the MailService.

public interface MailService {
  List<Mail> getMail();
}

And last, an interface for contracting an engine responsible for collecting Mail from multiple services, the MailEngine.

public interface MailEngine {
  List<Mail> getAllMail();
}

Implementations

Mail

The concrete Mail implementations were designed with a builder pattern for convenience and immutability.
The Gmail Mail implementation, GmailImpl:

public final class GmailImpl extends Mail {
  private final String setFrom;
  private final String setSubject;

  private GmailImpl(final String from, final String subject) {
    setFrom = from;
    setSubject = subject;
  }

  @Override
  public String from() {
    return setFrom;
  }

  @Override
  public String subject() {
    return setSubject;
  }

  @Override
  public MailSource source() {
    return MailSource.GMAIL;
  }

  public static GmailImpl.Builder builder() {
    return new GmailImpl.Builder();
  }

  public static final class Builder {
    private String prepFrom;
    private String prepSubject;

    public Builder from(final String setFrom) {
      prepFrom = setFrom;
      return this;
    }

    public Builder subject(final String setSubject) {
      prepSubject = setSubject;
      return this;
    }

    public GmailImpl build() {
      requireNonNull(emptyToNull(prepFrom), "from cannot be empty or null");
      requireNonNull(emptyToNull(prepSubject), "subject cannot be empty or null");

      return new GmailImpl(prepFrom, prepSubject);
    }
  }
}

The Micsorosft Mail implementation, MicrosoftImpl:

public final class MicrosoftImpl extends Mail {
  private final String setFrom;
  private final String setSubject;

  private MicrosoftImpl(final String from, final String subject) {
    setFrom = from;
    setSubject = subject;
  }

  @Override
  public String from() {
    return setFrom;
  }

  @Override
  public String subject() {
    return setSubject;
  }

  @Override
  public MailSource source() {
    return MailSource.MICROSOFT;
  }

  public static MicrosoftImpl.Builder builder() {
    return new MicrosoftImpl.Builder();
  }

  public static final class Builder {
    private String prepFrom;
    private String prepSubject;

    public Builder from(final String setFrom) {
      prepFrom = setFrom;
      return this;
    }

    public Builder subject(final String setSubject) {
      prepSubject = setSubject;
      return this;
    }

    public MicrosoftImpl build() {
      requireNonNull(emptyToNull(prepFrom), "from cannot be empty or null");
      requireNonNull(emptyToNull(prepSubject), "subject cannot be empty or null");

      return new MicrosoftImpl(prepFrom, prepSubject);
    }
  }
}

Mail Services

The Gmail MailService implementation:

public final class GmailService implements MailService {
  @Override
  public List<Mail> getMail() {
    //This is where the actual Gmail API access goes.
    //We'll fake a couple of emails instead.
    var firstFakeMail =
        GmailImpl.builder()
            .from("a.cool.friend@gmail.com")
            .subject("wanna get together and write some code?")
            .build();

    var secondFakeMail =
        GmailImpl.builder()
            .from("an.annoying.salesman@some.company.com")
            .subject("wanna buy some stuff?")
            .build();

    return List.of(firstFakeMail, secondFakeMail);
  }
}

The Microsoft MailService implementation:

public final class MicrosoftService implements MailService {
  @Override
  public List<Mail> getMail() {
    //This is where the actual Microsoft API access goes.
    //We'll fake a couple of emails instead.
    var firstFakeMail =
        MicrosoftImpl.builder()
            .from("my.boss@work.info")
            .subject("stop writing tutorials and get back to work!")
            .build();

    var secondFakeMail =
        MicrosoftImpl.builder()
            .from("next.door.neighbor@kibutz.org")
            .subject("do you have philips screwdriver?")
            .build();

    return List.of(firstFakeMail, secondFakeMail);
  }
}

Mail Engine

public final class RobustMailEngine implements MailEngine {
  private final Set<MailService> mailServices;

  public RobustMailEngine(final Set<MailService> setMailSerices) {
    mailServices = setMailSerices;
  }

  @Override
  public List<Mail> getAllMail() {
    return mailServices.stream().map(MailService::getMail).flatMap(List::stream).collect(toList());
  }
}

The Main App

This is the app itself, the MailCollectorApp:

public final class MailCollectorApp {
  private MailEngine engine;

  public MailCollectorApp(final MailEngine setEngine) {
    engine = setEngine;
  }

  public String getMail() {
    var ret = "No mail found.";
    if (!engine.getAllMail().isEmpty()) {
      ret = Joiner.on(System.lineSeparator()).join(engine.getAllMail());
    }
    return ret;
  }

  public static void main(final String... args) {
    var gmailService = new GmailService();
    var microsoftService = new MicrosoftService();

    var engine = new RobustMailEngine(Set.of(gmailService, microsoftService));

    var app = new MailCollectorApp(engine);

    System.out.println(app.getMail());
  }
}

Executing the main method will print:

Got Mail by GMAIL, from a.cool.friend@gmail.com, with the subject wanna get together and write some code?
Got Mail by GMAIL, from an.annoying.salesman@some.company.com, with the subject wanna buy some stuff?
Got Mail by MICROSOFT, from my.boss@work.info, with the subject stop writing tutorials and get back to work!
Got Mail by MICROSOFT, from next.door.neighbor@kibutz.org, with the subject do you have a star screwdriver?

This application uses the dependency injection design pattern. The dependencies are currently controlled by the main method, so it should be easy to incorporate Google Guice.

Incorporating Spring Context

Include maven dependency

First, let's add this to our pom.xml in the dependencies section:
Note that this version was the latest when this tutorial was written.

<dependency>
  <groupId>com.google.inject</groupId>
  <artifactId>guice</artifactId>
  <version>4.2.3</version>
</dependency>

Mark Inject

We need to tell Guice about the dependencies we want to be injected. Let's do this for the concrete engine class:

public final class RobustMailEngine implements MailEngine {
  private final Set<MailService> mailServices;

  @Inject
  public RobustMailEngine(final Set<MailService> setMailSerices) {
    mailServices = setMailSerices;
  }

  @Override
  public List<Mail> getAllMail() {
    return mailServices.stream().map(MailService::getMail).flatMap(List::stream).collect(toList());
  }
}

And for the app class:

public final class MailCollectorApp {
  private MailEngine engine;

  @Inject
  public MailCollectorApp(final MailEngine setEngine) {
    engine = setEngine;
  }

  public String getMail() {
    var ret = "No mail found.";
    if (!engine.getAllMail().isEmpty()) {
      ret = Joiner.on(System.lineSeparator()).join(engine.getAllMail());
    }
    return ret;
  }

  //...
}

Now, we need to instruct Guice on providing those dependencies.

Provide Dependencies

Let's create a Guice Module for configuring the bindings; multiple approaches exist to achieve that. For instance, using Guice's Provides Methods:

public final class DIModule extends AbstractModule {
  @Provides
  static Set<MailService> getServices() {
    return Set.of(new GmailService(), new MicrosoftService());
  }

  @Provides
  @Singleton
  static MailEngine getEngine(final Set<MailService> services) {
    return new RobustMailEngine(services);
  }
}

Another approach to configuring Guice bindings is using Guice's bind DSL:

public final class DIModule extends AbstractModule {
  @Override
  public void configure() {
    var listBinder = newSetBinder(binder(), MailService.class);
    listBinder.addBinding().toInstance(new GmailService());
    listBinder.addBinding().toInstance(new MicrosoftService());

    bind(MailEngine.class).to(RobustMailEngine.class).in(Scopes.SINGLETON);
  }
}

Both options will produce the same dependencies.

Guice's default scope is Non-Singleton. This means that the Set of GmailService and MicrosoftService will be created as new instances for every object that needs them.

On the other end, the RobustMailEngine will be bound as a Singelton.
Based on Guice's docs, that means that in Production stage it will be an Eager Singleton, and in Development stage it will be a Lazy Singleton.
Either way, it's going to be a Singleton, meaning our app will have only one instance of RobustMailEngine.

The default stage for Guice is Development stage, so we can expect a Lazy Singleton.

Note, RobustMailEngine is a dependency; it will be injected to whoever requests it, but as we configured earlier, it also needs dependencies for itself (the Set of MailService).

Guice will catch that and inject the set of mail services while instantiating the engine.

Update the app to use Guice

Getting back to our app. Let's update it to work with Guice:

public final class MailCollectorApp {
  private MailEngine engine;

  @Inject
  public MailCollectorApp(final MailEngine setEngine) {
    engine = setEngine;
  }

  public String getMail() {
    var ret = "No mail found.";
    if (!engine.getAllMail().isEmpty()) {
      ret = Joiner.on(System.lineSeparator()).join(engine.getAllMail());
    }
    return ret;
  }

  public static void main(final String... args) {
    var injector = Guice.createInjector(new DIModule());

    var app = injector.getInstance(MailCollectorApp.class);

    System.out.println(app.getMail());
  }
}

Let's analyze what's going on here...

Running the main method will create a Guice's injector, which is the context for all dependencies. As we create it with an instance of our DIModule class, the injector's context will have the following dependencies configured in it:

A Set of two MailService objects (GmailService and Microsoft Service).
A Singleton instance of MailEngine (RobustMailEngine).

As stated, in the Development stage the MainEngine is a Lazy Singleton, plus, we configured the Set of MailService as Non-Singleton. That means that there's nothing instantiated in the injector's context at this point.

The next step, asking the injector for an instance of MailCollectorApp, will accomplish the following:

Guice will pick up the constructor in MailCollectorApp, as it's the only constructor.
As the constructor is marked with @Inject, Guice will look in its context for a dependency of type MailEngine.
It will find the RobustMailEngine configured, which is a Lazy Singleton.
While trying to instantiate it, it will pick up its constructor, which is also marked with @Inject. And look for a suitable dependency with the type Set of MailService.
It will find the Set of GoogleService and MicrosoftService, which is a Non-Singleton.

After preparing the dependency graph, Guice will:

Create the set after instantiating both GmailService and MicrosoftService.
Instantiate the RobustMailEngine injecting the Set.
Instantiate the MailCollectorApp injecting the RobustMailEngine.

We then get our instance MailCollectorApp with everything we need, from which we invoke getMail to get all our Mail.

That's it, Guice in a nutshell.
😆

Now, Let's test the code.

Unit Tests

I will start by saying that when it comes to unit tests, if possible, I always prefer not to use DI Frameworks.

Unit tests are about testing small parts, units, of our application. We can avoid the overhead of creating the DI Context. We'll be better off simply instantiating the subject under test manually.

Let's move on to unit tests with a DI Framework for demonstration purposes only.

public final class MailCollectorAppTest extends AbstractModule {
  private MailService gmailServiceMock;
  private MailService microsoftServiceMock;
  private MailService thirdServiceMock;

  private MailCollectorApp sut;

  private Faker faker;

  @Override
  public void configure() {
    var listBinder = newSetBinder(binder(), MailService.class);
    listBinder.addBinding().toInstance(gmailServiceMock);
    listBinder.addBinding().toInstance(microsoftServiceMock);
    listBinder.addBinding().toInstance(thirdServiceMock);

    bind(MailEngine.class).to(RobustMailEngine.class).in(Scopes.SINGLETON);
  }

  @BeforeEach
  public void initialize() {
    faker = new Faker();

    gmailServiceMock = mock(MailService.class);
    microsoftServiceMock = mock(MailService.class);
    thirdServiceMock = mock(MailService.class);

    var injector = Guice.createInjector(this);

    sut = injector.getInstance(MailCollectorApp.class);
  }

  @Test
  @DisplayName(
      "make the services mocks return no mail and validate the return string as 'No mail found'")
  public void getMail_noMailExists_returnsNoMailFound() {
    willReturn(emptyList()).given(gmailServiceMock).getMail();
    willReturn(emptyList()).given(microsoftServiceMock).getMail();
    willReturn(emptyList()).given(thirdServiceMock).getMail();

    then(sut.getMail()).isEqualTo("No mail found.");
  }

  @Test
  @DisplayName(
      "make the services return legitimate mail and validate the return string as expected")
  public void getMail_foundMail_returnsExpectedString() {
    var mail1 =
        GmailImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();
    var mail2 =
        MicrosoftImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();
    var mail3 =
        MicrosoftImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();

    willReturn(List.of(mail1)).given(gmailServiceMock).getMail();
    willReturn(List.of(mail2, mail3)).given(microsoftServiceMock).getMail();
    willReturn(emptyList()).given(thirdServiceMock).getMail();

    then(sut.getMail().split(System.lineSeparator()))
        .containsOnly(mail1.toString(), mail2.toString(), mail3.toString());
  }
}

We created the injector with a different module than the one we've used for the app (DIModule).

In this case, we extended the test class to be a module in itself. Injecting our mocks instead of the real GmailService and MicrosoftService. We also added a third mail service, demonstrating how easy it is. 😁

The RobustMailEngine was not mocked because there was no real reason to do so, but it could have been easily replaced with a mock or a spy.

The test class behaved exactly like the main app, except for the services being mocks instead of real objects.

Dependency Injection in Java is easy - Part 1 - A Design Pattern

Tomer Figenblat — Sat, 08 Aug 2020 12:05:17 +0000

Part 1 - A Design Pattern

This is part of a multiple-part tutorial. This first part will explain the concept of Dependency Injection, a design pattern used for achieving Inversion of Control by writing loosely coupled code.

You can check out the code for this tutorial part in Github.

What is Inversion of Control

Let's visit the wiki definition, "Inversion of Control inverts the flow of control as compared to traditional control flow".

A traditional control flow is: 🅰️ ➡️ 🅱️ ➡️ 🅾️

🅰️ is the customer invoking 🅱️, the provider that uses 🅾️, the service for performing some kind of action.

🅱️ controls 🅾️
🅱️ depends on 🅾️
🅱️ and 🅾️ are tightly coupled

Inverting the flow of control means decoupleing 🅱️ from 🅾️ by giving the control of it to 🅰️. Changing the flow to: 🅰️ (🅾️) ➡️ 🅱️ ➡️ ❌

Now, 🅰️ is the customer, describing 🅾️ the service and invoking 🅱️ the provider for performing some action with the given service.

🅱️ doesn't control 🅾️
🅱️ doesn't depend on 🅾️
🅱️ and 🅾️ are loosely coupled

This is achieved using Dependency Injection.

What is Dependency Injection

Let's visit the wiki definition, "Dependency Injection is a technique in which an object receives other objects that it depends on."

So, Dependency Injection is a Design Pattern. Sounds less scary. Let's code.

Mail Collector App

Let's build an app pulling emails from both Gmail and Microsoft.

Contracts

An Enum called MailSource for categorizing the email source:

public enum MailSource {
  GMAIL,
  MICROSOFT;
}

An abstract class Mail for contracting mail objects.

public abstract class Mail {
  public abstract String from();

  public abstract String subject();

  public abstract MailSource source();

  @Override
  public String toString() {
    return String.format("Got mail by %s, from %s, with the subject %s", source(), from(), subject());
  }
}

An interface for contracting services responsible for pulling Mail from suppliers, the MailService.

public interface MailService {
  List<Mail> getMail();
}

And last, an interface for contracting an engine responsible for collecting Mail from multiple services, the MailEngine.

public interface MailEngine {
  List<Mail> getAllMail();
}

Implementations

Mail

The concrete Mail implementations were designed with a builder pattern for convenience and immutability.
The Gmail Mail implementation, GmailImpl:

public final class GmailImpl extends Mail {
  private final String setFrom;
  private final String setSubject;

  private GmailImpl(final String from, final String subject) {
    setFrom = from;
    setSubject = subject;
  }

  @Override
  public String from() {
    return setFrom;
  }

  @Override
  public String subject() {
    return setSubject;
  }

  @Override
  public MailSource source() {
    return MailSource.GMAIL;
  }

  public static GmailImpl.Builder builder() {
    return new GmailImpl.Builder();
  }

  public static final class Builder {
    private String prepFrom;
    private String prepSubject;

    public Builder from(final String setFrom) {
      prepFrom = setFrom;
      return this;
    }

    public Builder subject(final String setSubject) {
      prepSubject = setSubject;
      return this;
    }

    public GmailImpl build() {
      requireNonNull(emptyToNull(prepFrom), "from cannot be empty or null");
      requireNonNull(emptyToNull(prepSubject), "subject cannot be empty or null");

      return new GmailImpl(prepFrom, prepSubject);
    }
  }
}

The Micsorosft Mail implementation, MicrosoftImpl:

public final class MicrosoftImpl extends Mail {
  private final String setFrom;
  private final String setSubject;

  private MicrosoftImpl(final String from, final String subject) {
    setFrom = from;
    setSubject = subject;
  }

  @Override
  public String from() {
    return setFrom;
  }

  @Override
  public String subject() {
    return setSubject;
  }

  @Override
  public MailSource source() {
    return MailSource.MICROSOFT;
  }

  public static MicrosoftImpl.Builder builder() {
    return new MicrosoftImpl.Builder();
  }

  public static final class Builder {
    private String prepFrom;
    private String prepSubject;

    public Builder from(final String setFrom) {
      prepFrom = setFrom;
      return this;
    }

    public Builder subject(final String setSubject) {
      prepSubject = setSubject;
      return this;
    }

    public MicrosoftImpl build() {
      requireNonNull(emptyToNull(prepFrom), "from cannot be empty or null");
      requireNonNull(emptyToNull(prepSubject), "subject cannot be empty or null");

      return new MicrosoftImpl(prepFrom, prepSubject);
    }
  }
}

Mail Services

The Gmail MailService implementation:

public final class GmailService implements MailService {
  @Override
  public List<Mail> getMail() {
    //This is where the actual Gmail API access goes.
    //We'll fake a couple of emails instead.
    var firstFakeMail =
        GmailImpl.builder()
            .from("a.cool.friend@gmail.com")
            .subject("wanna get together and write some code?")
            .build();

    var secondFakeMail =
        GmailImpl.builder()
            .from("an.annoying.salesman@some.company.com")
            .subject("wanna buy some stuff?")
            .build();

    return List.of(firstFakeMail, secondFakeMail);
  }
}

The Microsoft MailService implementation:

public final class MicrosoftService implements MailService {
  @Override
  public List<Mail> getMail() {
    //This is where the actual Microsoft API access goes.
    //We'll fake a couple of emails instead.
    var firstFakeMail =
        MicrosoftImpl.builder()
            .from("my.boss@work.info")
            .subject("stop writing tutorials and get back to work!")
            .build();

    var secondFakeMail =
        MicrosoftImpl.builder()
            .from("next.door.neighbor@kibutz.org")
            .subject("do you have philips screwdriver?")
            .build();

    return List.of(firstFakeMail, secondFakeMail);
  }
}

Mail Engine

First, let's build the concrete MailEngine in a tightly coupled manner:

public final class TightMailEngine implements MailEngine {
  private final MailService gmailService;
  private final MailService microsoftService;

  public TightMailEngine() {
    gmailService = new GmailService();
    microsoftService = new MicrosoftService();
  }

  @Override
  public List<Mail> getAllMail() {
    return concat(
            gmailService.getMail().stream(),
            microsoftService.getMail().stream())
        .collect(toList());
  }
}

Notice TightMailEngine tightly depends on both GmailService and MicrosoftService. We can fix two things in this code with Dependency Injection.

Testing this code is hard, invoking the getAllMail method will invoke the real Gmail and Microsoft's services.
Adding another service will require modifications to TightMailEngine.

We can address testing by rewriting the engine loosely coupled as our first step into the Dependency Injection design pattern.

public final class LooseMailEngine implements MailEngine {
  private final MailService gmailService;
  private final MailService microsoftService;

  public LooseMailEngine(final MailService setGmailService, final MailService setMicrosoftService) {
    gmailService = setGmailService;
    microsoftService = setMicrosoftService;
  }

  @Override
  public List<Mail> getAllMail() {
    return concat(
            gmailService.getMail().stream(),
            microsoftService.getMail().stream())
        .collect(toList());
  }
}

The LooseMailEngine doesn't know, nor does it depend on, either GmailService or MicrosoftService. Instead, it expects them to be injected as constructor arguments. This is Inversion of control. The engine no longer has control of GmailService or MicrosoftService.

From a testing perspective, we can now easily inject mocks for both GmailService and MicrosoftService and prevent invocation of the real services.

Now, that we have leveraged dependency injection, we rewrite LooseMailEngine again and make it more robust:

public final class RobustMailEngine implements MailEngine {
  private final Set<MailService> mailServices;

  public RobustMailEngine(final Set<MailService> setMailSerices) {
    mailServices = setMailSerices;
  }

  @Override
  public List<Mail> getAllMail() {
    return mailServices.stream().map(MailService::getMail).flatMap(List::stream).collect(toList());
  }
}

Now, whoever invokes our engine has complete control of the service dependencies.
😎

The Main App

This is the app itself, the MailCollectorApp:

public final class MailCollectorApp {
  private MailEngine engine;

  public MailCollectorApp(final MailEngine setEngine) {
    engine = setEngine;
  }

  public String getMail() {
    var ret = "No mail found.";
    if (!engine.getAllMail().isEmpty()) {
      ret = Joiner.on(System.lineSeparator()).join(engine.getAllMail());
    }
    return ret;
  }

  public static void main(final String... args) {
    var engine = new RobustMailEngine(List.of(new GmailService(), new MicrosoftService()));

    var app = new MailCollectorApp(engine);

    System.out.println(app.getMail());
  }
}

Executing the main method will print:

Got mail by GMAIL, from a.cool.friend@gmail.com, with the subject wanna get together and write some code?
Got mail by GMAIL, from an.annoying.salesman@some.company.com, with the subject wanna buy some stuff?
Got mail by MICROSOFT, from my.boss@work.info, with the subject stop writing tutorials and get back to work!
Got mail by MICROSOFT, from next.door.neighbor@kibutz.org, with the subject do you have a star screwdriver?

Now, Let's test the code.

Unit Tests

We should always prefer to test smaller units of our code while writing unit tests. However, for brevity, we're testing the engine and the app combined while mocking the services:

public final class MailCollectorAppTest {
  private MailService gmailServiceMock;
  private MailService microsoftServiceMock;
  private MailService thirdServiceMock;

  private RobustMailEngine robustEngine;
  private MailCollectorApp sut;

  private Faker faker;

  @BeforeEach
  public void initialize() {
    faker = new Faker();

    gmailServiceMock = mock(MailService.class);
    microsoftServiceMock = mock(MailService.class);
    thirdServiceMock = mock(MailService.class);

    robustEngine =
        new RobustMailEngine(Set.of(gmailServiceMock, microsoftServiceMock, thirdServiceMock));
    sut = new MailCollectorApp(robustEngine);
  }

  @Test
  @DisplayName(
      "make the services mocks return no mail and validate the return string as 'No mail found'")
  public void getMail_noMailExists_returnsNoMailFound() {
    willReturn(emptyList()).given(gmailServiceMock).getMail();
    willReturn(emptyList()).given(microsoftServiceMock).getMail();
    willReturn(emptyList()).given(thirdServiceMock).getMail();

    then(sut.getMail()).isEqualTo("No mail found.");
  }

  @Test
  @DisplayName(
      "make the services return legitimate mail and validate the return string as expected")
  public void getMail_foundMail_returnsExpectedString() {
    var mail1 =
        GmailImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();
    var mail2 =
        MicrosoftImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();
    var mail3 =
        MicrosoftImpl.builder()
            .from(faker.internet().emailAddress())
            .subject(faker.lorem().sentence())
            .build();

    willReturn(List.of(mail1)).given(gmailServiceMock).getMail();
    willReturn(List.of(mail2, mail3)).given(microsoftServiceMock).getMail();
    willReturn(emptyList()).given(thirdServiceMock).getMail();

    then(sut.getMail().split(System.lineSeparator()))
        .containsOnly(mail1.toString(), mail2.toString(), mail3.toString());
  }
}

The next parts of this tutorial series are Part 2 - Google Guice and Part 3 - Spring Context. We'll introduce Dependency Injection Frameworks to the above code.

DEV Community: Tomer Figenblat

The developer's guide to Kubernetes Operators

TOC

What is an Operator?

What is a manager?

What is an API?

What are subresources?

What is a controller?

What are finalizers?

What is a scheme?

What are metrics?

What is a metrics server?

What is a health probe?

What is a webhook server?

What about versioning?

What is Helm?

What is OLM?

What is Kustomize?

Quick links

Development tools

Development toolbox

Useful packages

Common interfaces

Code markers

How to distribute workloads using Open Cluster Management

Basic Open Cluster Management

Setting up the example environment

Bootstrapping OCM

Initializing the hub cluster

Joining and accepting the managed clusters

Workload distribution across managed clusters

Grouping managed clusters

Selecting clusters from the set

Using labels to select clusters

Using ClusterClaims to select clusters

Using taints and tolerations for the selection

Fetching the selected cluster list

Prioritizing clusters for selection

Summarizing OCM basics

How to prevent computer overload with remote kind clusters

Why I researched tools to prevent computer overload

How to set up kind remote clusters

Bonus: Loading images to remote clusters

Tools that simplify Kubernetes

Protect secrets in Git with the clean/smudge filter

Clean and smudge your Git repository

Create the filter driver

One filter to rule them all

Configure a global filter

Conclusion

Testing Java with JUnit Pioneer

Making test code readable with JUnit Pioneer

Property-Based Matrix Testing in Java

Alexa Skills Testing with Java

Test your Java Alexa skills

Example usage

A launch request handler

An intent request handler handling MyNameIntent with a nameSlot

A session ended request handler

A utility class for constructing the skill

Now, let's test the skill

launch_request.json

my_name_intent.json

session_ended.json

Test cases testing the Nice to meet you skill

Migrating to Java Platform Module System

JPMS Migration Playground

Scenario Walkthrough

JPMS: foo as Automatic Module

Migrating foo

Create a module-info descriptor

Create a modular jar

Use the modular jar

Add the new module to the Modulepath

Create a new Classpath

Create a new Modulepath

Patch compiler with new Modulepah

Patch testing Classpath

My Ultimate Development Container image

My Ultimate Development Container Image

An intent request handler handling `MyNameIntent` with a `nameSlot`

Test cases testing the `Nice to meet you` skill