DEV Community: Emre Baykal

Building a Morpheus Plugin A Practical Walkthrough

Emre Baykal — Wed, 29 Apr 2026 20:10:27 +0000

Morpheus is a hybrid cloud management platform with a plugin SDK that lets you extend almost everything cloud integrations, custom reports, IPAM and DNS providers, backup integrations, UI tabs, global UI hooks. The SDK is documented in pieces, but the day-to-day workflow of going from "I have an idea" to "shadow JAR is running on my appliance" isn't laid out anywhere. This post walks through that workflow.

I'll use a plugin we recently built as the running example: a small enforcement utility that forces every Morpheus user to enable 2FA. If MFA isn't your problem, the example is incidental. What matters is the path.

What you can extend

The SDK exposes more than fifty provider types. A provider is your hook into Morpheus; Morpheus calls back into your code at well defined moments, and you pick the type that matches what you want to do.

A non exhaustive sample, mostly to give a feel for the catalog:

CloudProvider, ProvisionProvider — add a new cloud integration
BackupProvider — register a backup integration
ReportProvider — add a custom report type
AppTabProvider, ServerTabProvider, ClusterTabProvider, InstanceTabProvider, NetworkTabProvider — add a tab to a detail page
GlobalUIComponentProvider — inject HTML into every authenticated page render
TaskProvider — register a task type for the automation engine
CypherProvider — extend the secure-vault subsystem
GuidanceRecommendationProvider, AnalyticsProvider — feed dashboard widgets
GenericProvider — generic integration for things that don't fit elsewhere

The full list lives at morpheus-plugin-core/morpheus-plugin-api/src/main/java/com/morpheusdata/core/providers/. Picking the right one is the most important early decision; everything else flows from it.

Set up your reference material first

Before writing any code, clone these three repositories locally:

git clone --depth 1 --branch v1.2.x \
    https://github.com/HewlettPackard/morpheus-plugin-core.git
git clone --depth 1 \
    https://github.com/HewlettPackard/morpheus-docs.git
git clone --depth 1 \
    https://github.com/HewlettPackard/morpheus-openapi.git

morpheus-plugin-core is the source of the com.morpheusdata:morpheus-plugin-api library you'll compile against. Provider interfaces, abstract classes, model classes everything is here. When you want to know what a method returns or which fields a model exposes, grep this.

morpheus-docs is the source of the official Morpheus product documentation, currently published as HPE Morpheus VM Essentials on the HPE support portal. When you need to understand a feature from the user's perspective what does Morpheus call this concept, how does the UI organize it this is the reference.

morpheus-openapi is the OpenAPI 3.1 spec for Morpheus's REST API: 600+ endpoint files, 650+ schemas. If your plugin needs to call Morpheus over HTTP, or you need the canonical shape of a data model, this is faster than the live docs site.

Grep is fast; docs search is slow. A local clone pays off within the first hour.

A few web resources are worth bookmarking alongside the local clones:

developer.morpheusdata.com — Morpheus developer portal landing page; starter guides and links into the rest of the developer materials.
developer.morpheusdata.com/docs — plugin development handbook, organized by provider type and concept (Reports, Cloud, Tasks, UI Extensions, etc.).
developer.morpheusdata.com/api — plugin API Javadoc, useful when you want a browsable class hierarchy view instead of grepping the source.

Use them when the local clones don't have what you need; the local clones answer most questions faster.

Pick the right provider type

The methodology is straightforward:

Write down what you want Morpheus to do.
Translate that into a question: when does Morpheus need to call my code?
Find the provider whose method signatures answer that question.

For the MFA enforcer, the chain went:

"Force users without 2FA to enable it before they can use Morpheus."

"We need to be invoked on every page render, with the current user, so we can decide whether to render a blocking overlay."

Match: GlobalUIComponentProvider, which defines Boolean show(User, Account) and HTMLResponse renderTemplate(User, Account). Morpheus invokes both on every authenticated page render.

If we'd wanted a custom report, we'd have picked ReportProvider. If we'd wanted a tab on the instance detail page, InstanceTabProvider. The question-to-provider mapping becomes mechanical once you know the catalog.

One thing worth knowing up front: not every hook you might want exists. There is no LoginProvider and no AuthenticationProvider — you can't intercept the login flow itself. For the MFA enforcer that meant settling for GlobalUIComponentProvider, which runs after login. The SDK won't always give you exactly what you want; pick the closest match and design around its constraints.

Build environment

The plugin API targets Java 8, but Gradle 7.x needs JDK 11 to build. The shadow plugin used by older Morpheus samples (com.bmuschko.gradle-plugin-shadow 2.0.3) is Gradle 6.x only — use the maintained com.github.johnrengelman.shadow 7.1.2 instead.

On Linux:

curl -fsSL "https://get.sdkman.io" | bash
source "$HOME/.sdkman/bin/sdkman-init.sh"

sdk install java 11.0.22-tem
sdk install gradle 7.6.4

cd my-plugin
gradle wrapper --gradle-version 7.6.4 --distribution-type bin
./gradlew shadowJar

A minimal build.gradle:

plugins {
    id 'java'
    id 'groovy'
    id 'com.github.johnrengelman.shadow' version '7.1.2'
}

group   = 'com.morpheusdata'
version = '1.0.0'

sourceCompatibility = '11'
targetCompatibility = '11'

repositories { mavenCentral() }

dependencies {
    compileOnly 'com.morpheusdata:morpheus-plugin-api:0.15.4'
    compileOnly 'org.codehaus.groovy:groovy-all:3.0.11'
    compileOnly 'io.reactivex.rxjava3:rxjava:3.1.5'
    compileOnly 'org.slf4j:slf4j-api:1.7.26'
}

shadowJar {
    manifest {
        attributes(
            'Plugin-Class'  : 'com.example.myplugin.MyPlugin',
            'Plugin-Version': archiveVersion.get()
        )
    }
}

compileOnly is not optional. Morpheus's runtime already provides these libraries; bundling them into your shadow JAR causes classpath conflicts that surface as NoSuchMethodError at load time.

Bumping the version field on every revision is worth doing — the version becomes part of the JAR filename, so you can confirm which build is loaded by glancing at the plugin list in the Morpheus UI.

Anatomy of a plugin

Three pieces.

The Plugin class is your entry point, registered in the JAR manifest as Plugin-Class. It registers your providers in initialize():

class MfaEnforcerPlugin extends Plugin {
    @Override String getCode() { 'morpheus-mfa-enforcer-plugin' }

    @Override void initialize() {
        setName("MFA Enforcer")
        def provider = new MfaEnforcerProvider(this, morpheus)
        pluginProviders.put(provider.code, provider)
    }
}

The Provider class is where the work lives. Always extend the matching Abstract*Provider instead of implementing the interface directly. The abstract classes set up the renderer with the correct classpath prefix, register handlebars helpers (asset, nonce, i18n), and wire up things you don't want to wire up yourself.

@Slf4j
class MfaEnforcerProvider extends AbstractGlobalUIComponentProvider {
    Plugin plugin
    MorpheusContext morpheusContext

    MfaEnforcerProvider(Plugin plugin, MorpheusContext ctx) {
        this.plugin = plugin
        this.morpheusContext = ctx
    }

    @Override Boolean show(User user, Account account) {
        if (user == null || account == null) return false
        if (isMasterTenantAdmin(user, account)) return false
        return !isUsing2FA(user)
    }

    @Override HTMLResponse renderTemplate(User user, Account account) {
        def model = new ViewModel<>()
        model.object = [userSettingsUrl: '/user-settings']
        return getRenderer().renderTemplate("hbs/mfaEnforcer", model)
    }

    @Override MorpheusContext getMorpheus() { morpheusContext }
    @Override Plugin getPlugin() { plugin }
}

The getMorpheus() and getPlugin() overrides are required by PluginProvider and need to be explicit because Groovy's auto-generated getters use the field name (getMorpheusContext, not getMorpheus).

Resources live under src/main/resources/. Handlebars templates go in renderer/hbs/. The renderer's classpath prefix is renderer, so getRenderer().renderTemplate("hbs/mfaEnforcer", model) resolves to src/main/resources/renderer/hbs/mfaEnforcer.hbs.

For inline <style> and <script> tags, use the {{nonce}} helper:

<div id="mfa-shield" role="dialog" aria-modal="true">
    <style nonce="{{nonce}}">
        #mfa-shield {
            position: fixed; inset: 0; z-index: 2147483647;
            background: rgba(15, 23, 42, 0.75);
            display: flex; align-items: center; justify-content: center;
        }
        /* ... */
    </style>
    <div class="card">
        <h1>Two-Factor Authentication Required</h1>
        <a class="cta" href="{{userSettingsUrl}}" target="_top">
            Go to User Settings → Enable 2FA
        </a>
    </div>
</div>

Without the nonce, Morpheus's Content-Security-Policy header blocks the styles and the script silently. There's no console error, no log entry, no visible failure mode — the page just renders without your component. Easy hours to lose.

The example, end-to-end

A quick note on why this plugin exists, for the curious. Morpheus supports two-factor authentication out of the box, but only as an opt-in toggle on each user's profile there is no appliance wide setting to require it. The product documentation says so explicitly: "There is not currently a way for an administrator to enforce the use of two-factor authentication appliance wide." As an MSP we couldn't rely on every sub-tenant user enabling 2FA on their own, so we wrote a small plugin to close the gap.

MFA enforcer's full source is at emrbaykal/morpheus-plugin-test/morpheus-mfa-enforcer. It's roughly 200 lines split across four files:

morpheus-mfa-enforcer/
├── build.gradle                                          # project config
└── src/main/
    ├── groovy/com/morpheusdata/mfaenforcer/
    │   ├── MfaEnforcerPlugin.groovy                      # entry point, registers provider
    │   └── MfaEnforcerProvider.groovy                    # show() + renderTemplate() + state lookup
    └── resources/renderer/hbs/
        └── mfaEnforcer.hbs                               # overlay HTML + CSS + small JS

The decision logic in show() reads the user's 2FA flag from the Morpheus database through MorpheusReportService.getReadOnlyDatabaseConnection(). The result is cached in a static ConcurrentHashMap with a short TTL — show() is hot, called on every authenticated page render, and uncached SELECTs would generate noticeable load and contention against Morpheus's own writes.

The visible result, in five steps:

A user logs in:

The dashboard loads, but the overlay covers it immediately:

The CTA hard-navigates to /user-settings, where the standard Morpheus 2FA enable flow runs (password confirmation, QR code, authenticator app):

After enabling, the user-settings page reflects the new state:

On the next login, Morpheus prompts for the 2FA code from the authenticator:

And the dashboard loads cleanly the overlay is gone, because show() now reads is_using2fa = true:

Build, package, deploy

./gradlew shadowJar
# build/libs/morpheus-mfa-enforcer-1.0.10-all.jar

Then in the Morpheus UI: Administration → Integrations → Plugins → Choose File. Upload the shadow JAR. The plugin loads on every appliance node, no daemon restart required.

The plugin list in that section shows the version from the JAR filename. Combined with version-bump-on-every-revision, that's all the deployment tracking you need for a small plugin.

A few practical notes

A handful of things that aren't obvious from reading the SDK:

Always extend the matching Abstract*Provider. The abstract classes do non-trivial work — registering handlebars helpers, configuring the renderer's classpath prefix, wiring up locale support. Implementing the bare interface is reinventing this work, and the omissions tend to manifest as silent failures (template not found, CSS blocked by CSP) that are time-consuming to diagnose.

Be careful with hot-path DB access. Any provider invoked per page render needs caching. Without it you add latency to every authenticated request and, in the worst case, contend with Morpheus's own writes via InnoDB share locks. A small ConcurrentHashMap with a short TTL plus an explicit sql.commit() after each SELECT keeps the plugin a good citizen.

/api/* endpoints require an OAuth2 bearer token. Browser session cookies aren't accepted. If your plugin tries to fetch them from inline JavaScript in a template, you'll get 401. For state lookups, do them server-side via the database or via plugin-side service calls.

When something doesn't work, check the Morpheus daemon log first. Browser console second. Plugin failures often produce no console output at all — the symptoms are silent CSP blocks, silent SPA click intercepts, silent fail-open paths. The daemon log is more honest.

Closing

The plugin SDK is more capable than the documentation makes obvious. Once you have the three reference repositories cloned, the build environment set up, and the right Abstract*Provider class identified, the actual work moves quickly.

The same pattern applies to password complexity rules, IP-based access restrictions, role-aware session timeouts, and other guardrails the platform doesn't ship with: pick the provider that matches your hook, extend the abstract class, ship the shadow JAR.

If you've built something interesting with the SDK, drop a link in the comments.

Setting Up Disaster Recovery for HPE Morpheus Enterprise with MySQL InnoDB ClusterSet Step by Step

Emre Baykal — Mon, 20 Apr 2026 18:34:42 +0000

HPE Morpheus Enterprise Disaster Recovery with MySQL InnoDB ClusterSet: A Production Grade Guide

In a previous build out we deployed two separate MySQL InnoDB Clusters, one on the Master site and one on the DR site, as the backing database for our Morpheus Enterprise platform. This guide is the disaster recovery follow up: we merge those two clusters into a single InnoDB ClusterSet so Morpheus Enterprise keeps serving traffic even if an entire site is lost. It covers TLS encrypted async replication between the sites, re bootstrapping MySQL Router so the Morpheus app layer doesn't need any connection string changes, and both planned switchovers and emergency failovers, including the nasty errant GTID cleanup that most guides skip. Every step is executed without interrupting live Morpheus traffic.

Introduction: Why InnoDB ClusterSet?

MySQL InnoDB Cluster gives you solid high availability inside a single data center. Group Replication handles node failures transparently. What it doesn't give you is protection against a site level disaster: if the DC goes dark, a fiber gets cut, or a regional outage hits, your cluster vanishes with it.

InnoDB ClusterSet fills that gap:

A Primary Cluster serves write traffic (Master Site).
A Replica Cluster is fed via async replication (DR Site).
MySQL Router picks up topology changes automatically, so there are no application side connection string changes after failover.
Planned switchovers give you zero data loss; emergency failovers keep RPO in the single digit seconds range.

This article walks through the exact steps I run in production, with the error messages you will hit, the security posture you should aim for, and the operational tips most tutorials skip. It is specifically written to layer ClusterSet on top of two existing InnoDB Clusters rather than a greenfield install.

One caveat worth stating up front: ClusterSet is not a replacement for backups. Logical corruption (DROP TABLE, a bad migration, an application bug) replicates to DR just fine. You still need MySQL Enterprise Backup or Percona XtraBackup on an independent schedule.

Starting Point and Target Architecture

Current State

Component	Master Site	DR Site
InnoDB Cluster	Installed and running	Installed (will be dropped)
MySQL Router	Registered and active	Registered (will be re bootstrapped)
Database	Preserved (source of truth)	Doesn't matter (will be overwritten by clone)

Requirements and Pre-Flight Checks

Software Versions

Component	Version	Notes
MySQL Server	8.0.27+ (8.4 LTS recommended)	Identical version across every node
MySQL Shell	Same major as server	Required on every node
MySQL Router	8.0.x (8.2+ for R/W splitting)	At least one instance per site

Firewall Port List

Cross-site (Master site ↔ DR site, both directions):

Port	Protocol	Service	Purpose
3306	TCP	MySQL Classic	ClusterSet async replication channel (`clusterset_replication`) + AdminAPI

ClusterSet uses standard async replication on port 3306 between the primary and replica clusters. Group Replication's internal port (33061) is intra cluster only and does not carry cross site traffic.

Intra cluster (within the same site, between GR members):

Port	Protocol	Service	Purpose
3306	TCP	MySQL Classic	Client + recovery donor connections
33061	TCP	Group Replication	Inter-node GR `local_address` (configurable via `group_replication_local_address`)
33062	TCP	XCom (recommended free)	Used when `communicationStack: 'XCOM'`. On the default `MYSQL` stack it isn't required, but reserving it avoids surprises if you ever switch stacks

Operator workstation → any MySQL node:

Port	Protocol	Purpose
3306	TCP	`mysqlsh` classic connection
33060	TCP	`mysqlsh` X Protocol (default for newer Shells)

A common mistake: port 33060 is not required between MySQL nodes (cross site or intra cluster). It is only needed from the operator/DBA workstation. Leaving it open cross site is unnecessary attack surface.

Application → MySQL Router:

Port	Protocol	Purpose
6446	TCP	Classic R/W, routes to primary
6447	TCP	Classic R/O, load-balanced across secondaries
6448 / 6449	TCP	X Protocol R/W and R/O (optional)
6450	TCP	R/W splitting endpoint (MySQL Router 8.2+, optional)

Verify connectivity before you start:

# Cross-site: only 3306 is required
nc -zv <remote_site_node_ip> 3306

# Intra-cluster (within same site): 3306 and 33061
nc -zv <local_peer_ip> 3306
nc -zv <local_peer_ip> 33061

Server-Level Consistency Checks (often forgotten)

Any of the following drifting between Master and DR will eventually break replication or corrupt query semantics. Run on one node per cluster and diff the output.

-- Character set and collation
SHOW VARIABLES LIKE 'character_set_server';
SHOW VARIABLES LIKE 'collation_server';

-- Timezone (critical for TIMESTAMP columns)
SELECT @@global.time_zone, @@global.system_time_zone;

-- SQL mode
SELECT @@global.sql_mode;

-- Lower case table names (cannot be changed after initialization)
SELECT @@global.lower_case_table_names;

-- Clone plugin availability (required for recoveryMethod: 'clone')
SHOW PLUGINS WHERE Name = 'clone';
-- Expected: Status = ACTIVE. If missing:
-- INSTALL PLUGIN clone SONAME 'mysql_clone.so';

Watch out for lower_case_table_names. If Master and DR were initialized with different values, replication will break on any mixed case table name. The value is set at mysqld --initialize time and cannot be changed afterwards, so the only fix is a full re init of the divergent side.

Step 1: Configure `server_id` and `report_host` on the Existing Cluster

ClusterSet requires every MySQL node to have a unique server_id (across both sites) and a report_host that advertises its real address. These parameters don't take effect until MySQL restarts, so on a live cluster you perform a rolling restart. Never shut all nodes down at once.

Critical: restart secondaries first and the primary last. Otherwise you risk losing quorum.

1.1 Inspect Current Values

mysql -u clusterAdmin -p \
  -e "SELECT @@server_id, @@report_host, @@report_port, @@hostname;"

Target values:

Node	server_id	report_host
master_node1 (Primary)	1	master_node1 IP
master_node2 (Secondary)	2	master_node2 IP
master_node3 (Secondary)	3	master_node3 IP
dr_node1	4	dr_node1 IP
dr_node2	5	dr_node2 IP
dr_node3	6	dr_node3 IP

Why this matters: server_id must be unique across the entire ClusterSet, not just per cluster. Two nodes sharing an id will drop replication events or create a silent data divergence.

1.2 Edit `my.cnf`

[mysqld]
# ── Required for ClusterSet ──────────────────────────────────────
server_id                  = 1             # Must be UNIQUE across BOTH sites
report_host                = 10.10.1.11    # Real IP of this node
report_port                = 3306

# ── Should already be present on an InnoDB Cluster ───────────────
gtid_mode                  = ON
enforce_gtid_consistency   = ON
binlog_format              = ROW
log_replica_updates        = ON            # Renamed from log_slave_updates in 8.0.26
plugin_load_add            = group_replication.so

# ── Auto-rejoin after restart (optional but recommended) ─────────
loose-group_replication_start_on_boot = ON

A note on hostnames: use IP addresses for report_host. Hostname based setups frequently hit Server address configuration error when DNS is inconsistent.

1.3 Rolling Restart

The sub steps below follow the safe order: identify primary, restart each secondary in turn, restart primary last, move the primary role back to master_node1, then verify Router.

1.3.1 Identify the Primary

mysqlsh clusterAdmin@master_node1:3306
var cluster = dba.getCluster()
cluster.status()

The node with memberRole: "PRIMARY" is restarted last.

1.3.2 Restart Secondaries One at a Time

# A) (Optional) Gracefully remove from cluster
mysqlsh clusterAdmin@<secondary_node>:3306
# >> dba.getCluster().removeInstance('<secondary_node>:3306', {force: false})

# B) Restart MySQL
systemctl restart mysqld

# C) Verify new parameters
mysql -u clusterAdmin -p -e "SELECT @@server_id, @@report_host;"

# D) Re-add
mysqlsh clusterAdmin@master_node1:3306
# >> dba.getCluster().addInstance('<secondary_node>:3306', {recoveryMethod: 'incremental'})

Shortcut: with loose group_replication_start_on_boot = ON, a plain systemctl restart mysqld is enough. The node rejoins on its own within 30 to 60 seconds.

1.3.3 Restart the Primary Last

# Confirm every secondary is ONLINE first
systemctl restart mysqld

When mysqld exits, Group Replication performs a view change and promotes a healthy secondary (5–15 seconds).

1.3.4 Move the Primary Role Back

mysqlsh clusterAdmin@<any_node>:3306
dba.getCluster().setPrimaryInstance('master_node1:3306')

This is a coordinated primary handover, not an election. The current primary drains in flight transactions, sets the target node writable, and transfers the role. Router picks up the change via its metadata TTL (5 to 15 seconds in practice).

1.3.5 Verify Router Routing

mysql -h <router_ip> -P 6446 -u clusterAdmin -p \
  -e "SELECT @@hostname, @@report_host;"

Output should show master_node1.

1.4 Full Verification

Check	Expected
`MEMBER_STATE`	`ONLINE` for every node
`@@server_id`	Unique (1–6)
`@@report_host`	Real IP
`cluster.status()`	`statusText: "Cluster is ONLINE and can tolerate up to ONE failure"`

Step 2: Verify Server Certificates Are Present (Pre-Flight for TLS)

Cross-site async replication without TLS is a serious exposure in production. Before you create the ClusterSet, confirm that every node already has working server certificates so the replication channel can negotiate TLS as soon as it's created in §4.1. The actual enforcement option (clusterSetReplicationSslMode) is set at ClusterSet creation time, not here.

2.1 Confirm Certificates Exist

MySQL auto-generates self signed certs under datadir on first init. For production, replace them with certs issued by your internal CA. Verify they exist on every node:

SHOW VARIABLES LIKE 'ssl_%';
-- ssl_ca, ssl_cert, ssl_key should all point to valid files

Test TLS is actually offered:

mysql -u clusterAdmin -p -h master_node1 \
  --ssl-mode=REQUIRED \
  -e "SHOW STATUS LIKE 'Ssl_cipher';"
# Ssl_cipher should be non-empty (e.g., TLS_AES_256_GCM_SHA384)

2.2 How TLS Is Enforced (Preview of §4.1)

TLS on the ClusterSet replication channel is controlled by the clusterSetReplicationSslMode option, set at ClusterSet creation time via cluster.createClusterSet() (see §4.1). Possible values:

Value	Behavior
`AUTO` (default)	Enables encryption when the server supports it, disables otherwise
`REQUIRED`	Enables encryption for all ClusterSet replication channels
`DISABLED`	Disables encryption. Avoid in production
`VERIFY_CA` (Shell 8.0.33+)	Like `REQUIRED`, plus verifies the server certificate against the CA
`VERIFY_IDENTITY` (Shell 8.0.33+)	Like `VERIFY_CA`, plus verifies the certificate identity matches the server hostname

The internal replication user (typically mysql_innodb_cs_<hex>) is created by AdminAPI and the channel is configured to match this option. Optionally you can also restrict the account server-side post-creation as defense-in-depth:

-- Defense in depth; the channel is already configured per clusterSetReplicationSslMode
SELECT User, Host FROM mysql.user WHERE User LIKE 'mysql_innodb_cs_%';
ALTER USER 'mysql_innodb_cs_xxxx'@'<scoped_host>' REQUIRE SSL;

TLS version note: MySQL 8.0 negotiates TLSv1.2 and TLSv1.3 by default. Pinning to tls_version = TLSv1.3 is good hardening only when every client supports it; keeping TLSv1.2,TLSv1.3 is the safer default.

Step 3: Remove the Existing DR Cluster

The legacy InnoDB Cluster on DR cannot be added to a ClusterSet as a replica. It has to be fully dismantled first.

Data loss warning: this wipes DR. Clone re-seeds from Master afterwards. Confirm nothing on DR needs to be preserved before continuing.

3.1 Method A: `dissolve()` (Recommended)

mysqlsh clusterAdmin@dr_node1:3306

var oldCluster = dba.getCluster()
oldCluster.status()
oldCluster.dissolve({force: true})

dissolve() connects to every node, stops Group Replication, and drops the mysql_innodb_cluster_metadata schema. force: true lets it proceed even with OFFLINE nodes.

Verify on every DR node:

SELECT @@group_replication_group_name;                  -- empty
SHOW STATUS LIKE 'group_replication_primary_member';    -- empty

dba.getCluster()   // should throw

3.2 Method B: `dropMetadataSchema()` (Fallback)

If dissolve() fails:

mysqlsh clusterAdmin@dr_node1:3306

\sql
STOP GROUP_REPLICATION;
-- RESET REPLICA ALL is a no-op for a pure InnoDB Cluster (GR uses its own
-- group_replication_recovery channel, not SQL-level replication). It's only
-- useful here if the node was *previously* a ClusterSet replica or had some
-- manual async channel configured. Skip unless `SHOW REPLICA STATUS\G` shows
-- a channel. Running it on a clean node is harmless but unnecessary.
-- RESET REPLICA ALL;
\js

dba.dropMetadataSchema({force: true})

Repeat on dr_node2 and dr_node3. If you still see errant GTIDs blocking clone later, a controlled reset is sometimes necessary:

-- Use only if you're 100% sure DR data is disposable
RESET MASTER;                        -- MySQL 8.0.x
-- RESET BINARY LOGS AND GTIDS;      -- MySQL 8.4+ (new spelling; RESET MASTER is deprecated there)

3.3 Clean Up the DR Router

systemctl stop mysqlrouter
rm -rf /etc/mysqlrouter/*
rm -rf /var/lib/mysqlrouter/*

3.4 Confirm DR Nodes Are Standalone

mysqlsh clusterAdmin@<dr_nodeX>:3306
dba.getCluster()                                               // ERROR expected
dba.checkInstanceConfiguration('clusterAdmin@<dr_nodeX>:3306') // "ok"

Step 4: Create the ClusterSet and Join DR

4.1 Create ClusterSet from Master (TLS hardened)

Both TLS enforcement and the replication user host scope are configured at ClusterSet creation time. They belong on cluster.createClusterSet(), not on createReplicaCluster() later. Setting them here means every replica cluster you add inherits the secure defaults.

Assumed naming convention used throughout this guide (substitute your own names if different):

Primary cluster name (created earlier when you ran dba.createCluster('morpheus cluster') on Master): morpheus cluster

ClusterSet name (created in this step): MorpheusClusterSet

Replica cluster name (created in §4.2): MorphDrCluster

You can confirm the primary cluster name with dba.getCluster().getName() before proceeding.

mysqlsh clusterAdmin@master_node1:3306

var cluster = dba.getCluster()
// Sanity check: this should return 'morpheus cluster' in this guide
cluster.getName()

var clusterSet = cluster.createClusterSet('MorpheusClusterSet', {
  clusterSetReplicationSslMode: 'REQUIRED',        // Force TLS on the ClusterSet replication channel
  replicationAllowedHost: '10.0.0.0/8'             // MUST cover BOTH primary and replica subnets (see note below)
})

clusterSet.status()

Version requirements: clusterSetReplicationSslMode is available in MySQL Shell 8.0.27 and later. The values VERIFY_CA and VERIFY_IDENTITY (stronger than REQUIRED) were added in 8.0.33. The replicationAllowedHost option requires MySQL Shell 8.0.28 or later.

⚠️ replicationAllowedHost is bidirectional. Per the official docs, the host pattern you set must be reachable from nodes in both the primary and replica clusters. The internal replication user (mysql_innodb_cs_*) is granted with this host pattern and will be used by:

The replica cluster when it connects to the primary (normal-state replication)

The original primary's nodes if they later need to connect to a new primary after failover or switchover

Scoping the pattern to only the DR subnet (for example 10.20.0.0/24) will work as long as the primary never changes, but setPrimaryCluster() or forcePrimaryCluster() will break replication for the old primary's nodes because they can't satisfy the host grant. Use a CIDR or wildcard that covers every MySQL node in every cluster, for example 10.0.0.0/8 covering both sites, or a pair of wildcards like 10.10.%.% combined with 10.20.%.%.

Host pattern syntax: per the official docs, replicationAllowedHost accepts CIDR notation (for example 192.0.2.0/24). The MySQL Host column also accepts wildcards like 10.%.%.% and netmask form 10.20.0.0/255.255.0.0 if your network requires those. Avoid '%' in production, since it lets the replication user connect from any IP.

Already created? If you see MYSQLSH 51601 — function is not available … already belongs to a ClusterSet, fetch the existing ClusterSet:
var clusterSet = dba.getClusterSet()
// To change the host scope after the fact:
clusterSet.setOption('replicationAllowedHost', '10.0.0.0/8')

4.2 Add DR as Replica Cluster

TLS and host scope were already set at the ClusterSet level in §4.1, so the replica cluster inherits them automatically. Here you only need the provisioning options that belong on createReplicaCluster():

var clusterSet = dba.getClusterSet()

var replicaCluster = clusterSet.createReplicaCluster(
  'clusterAdmin@dr_node1:3306',
  'MorphDrCluster',
  {
    recoveryMethod: 'clone',     // AUTO | CLONE | INCREMENTAL
    recoveryProgress: 1,         // 0 = silent, 1 = static, 2 = progress bars
    timeout: 600                 // Seconds to wait for post-provisioning sync
    // communicationStack: 'MYSQL'   // Default on 8.0.27+; set only to pin explicitly
  }
)

What's actually happening under the hood, per the official docs: createReplicaCluster() creates the clusterset_replication asynchronous channel, generates an internal replication user with a random password, and configures encryption on the channel according to the clusterSetReplicationSslMode you set on the ClusterSet in §4.1. It also sets skip_replica_start = ON and super_read_only = ON, and enables mysql_start_failover_channels_if_primary for async failover on the channel.

Add remaining DR nodes:

replicaCluster.addInstance('clusterAdmin@dr_node2:3306', {recoveryMethod: 'clone'})
replicaCluster.addInstance('clusterAdmin@dr_node3:3306', {recoveryMethod: 'clone'})

Note on cloneDonor: by default AdminAPI picks a secondary from the primary cluster as clone donor (and falls back to the primary if no secondary is available). For cross-WAN clones of large datasets, consider pinning an explicit donor with cloneDonor: 'host:port' to control which node sources the snapshot, and therefore which link the traffic flows through.

4.3 Common Errors

Error	Cause	Fix
`Server address configuration error`	`report_host` missing/wrong	Add to `my.cnf`, restart mysqld
`Target instance already part of an InnoDB Cluster`	DR not cleaned up	Re-run Step 3
`Errant GTIDs detected`	Old transactions on DR	Use `recoveryMethod: 'clone'` (overwrites)
`Can't connect to primary cluster`	Firewall / `report_host`	Test with `nc -zv`
`clone plugin not installed`	Missing plugin	`INSTALL PLUGIN clone SONAME 'mysql_clone.so';` on both sides
`Access denied for user 'mysql_innodb_cs_*'`	TLS mismatch or host restriction	Verify `replicationAllowedHost` matches donor IP

Step 5: Re-bootstrap MySQL Router

DR's metadata was rebuilt, so Router on both sites must re-learn the new topology.

5.1 DR Router

Always bootstrap against a Master site node. Router reads metadata from the primary.

systemctl stop mysqlrouter
rm -rf /etc/mysqlrouter/*
rm -rf /var/lib/mysqlrouter/*

mysqlrouter --bootstrap clusterAdmin@master_node1:3306 \
  --account routeruser \
  --user=mysqlrouter \
  --conf-use-gr-notifications \
  --force
# Note: consider dropping --disable-rest if you want metrics/health endpoints

systemctl start mysqlrouter

5.2 Master Router

systemctl stop mysqlrouter

mysqlrouter --bootstrap clusterAdmin@master_node1:3306 \
  --account routeruser \
  --user=mysqlrouter \
  --conf-use-gr-notifications \
  --force

systemctl start mysqlrouter

--conf-use-gr-notifications makes Router react to GR state changes without waiting for a metadata TTL, which meaningfully shortens failover detection.

5.3 Keyring Error (`Keyring decryption failed`)

systemctl stop mysqlrouter

find / -name '*.keyring' 2>/dev/null
rm -f /var/lib/mysqlrouter/*.keyring
rm -f /var/lib/mysqlrouter/keyring*
rm -f /etc/mysqlrouter/*.key

mysqlrouter --bootstrap clusterAdmin@master_node1:3306 \
  --account routeruser --user=mysqlrouter --force

5.4 `mysql_native_password` Warning

Failed changing the authentication plugin: mysql_native_password is deprecated is a warning, not an error. In MySQL 8.4 the plugin is disabled by default. Upgrade the account cleanly:

ALTER USER 'routeruser'@'%' IDENTIFIED WITH caching_sha2_password BY '<password>';

Step 6: Verification and Monitoring

6.1 ClusterSet Health

mysqlsh clusterAdmin@master_node1:3306
var clusterSet = dba.getClusterSet()
clusterSet.status({extended: 1})

Expected:

{
  "primaryCluster": "morpheus-cluster",
  "status": "HEALTHY",
  "replicaClusters": {
    "MorphDrCluster": {
      "clusterRole": "REPLICA",
      "clusterSetReplicationStatus": "OK",
      "transactionSetConsistencyStatus": "OK"
    }
  }
}

If transactionSetConsistencyStatus is anything other than OK (especially INCONSISTENT), stop and investigate before letting traffic continue.

6.2 Replication Channel Queries

A handful of queries cover the vast majority of troubleshooting on a running ClusterSet:

-- 1) ClusterSet replication channel status (run on DR primary)
SELECT CHANNEL_NAME, SERVICE_STATE, LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE
FROM performance_schema.replication_connection_status
WHERE CHANNEL_NAME = 'clusterset_replication';

-- 2) Applier worker status (look for LAST_ERROR on any worker)
SELECT CHANNEL_NAME, WORKER_ID, SERVICE_STATE, LAST_ERROR_NUMBER, LAST_ERROR_MESSAGE
FROM performance_schema.replication_applier_status_by_worker
WHERE CHANNEL_NAME = 'clusterset_replication';

-- 3) GTID gap check: the DR executed set should be a subset of Master's
SELECT @@global.gtid_executed;    -- run on both, compare

-- 4) Classic lag view (still useful for a quick eyeball)
SHOW REPLICA STATUS\G
-- Seconds_Behind_Source (8.0.22+) / Seconds_Behind_Master (older)

-- 5) Group Replication member view
SELECT MEMBER_HOST, MEMBER_ROLE, MEMBER_STATE
FROM performance_schema.replication_group_members
ORDER BY MEMBER_ROLE;

6.3 Router Registration and Ports

clusterSet.listRouters()

Router	Hostname	R/W Port	R/O Port
Master Router	`<master_server>`	6446	6447
DR Router	`<dr_server>`	6446	6447

ss -tlnp | grep mysqlrouter
# LISTEN  0  70  0.0.0.0:6446   # R/W
# LISTEN  0  70  0.0.0.0:6447   # R/O

6.4 Metrics to Pipe Into Prometheus

⚠️ Exporter version caveat: the metric names below are indicative and follow prometheus/mysqld_exporter historical naming. Metric names have changed across versions (e.g. slave → replica in newer releases, and some Group Replication metrics moved between collector flags). Before copy-pasting into your alerts, verify the exact names against the exporter version you run. Query your Prometheus with {__name__=~"mysql_.*(replica|slave).*"} and {__name__=~"mysql_.*group_replication.*"} and adjust. You may also need to enable collectors like --collect.slave_status / --collect.replica_status and --collect.perf_schema.replication_group_member_stats.

Metric (indicative name; verify against your exporter)	Threshold	Action
`mysql_slave_status_seconds_behind_master` (or `mysql_replica_status_seconds_behind_source` on newer exporters)	> 10 s sustained	Check WAN bandwidth, DR apply throughput
`mysql_perf_schema_replication_group_member_info{member_role="PRIMARY"}` or equivalent	Must match expected primary	Alert on unexpected primary change
`mysql_up` on any node	= 0 for > 60 s	Page on-call
`transactionSetConsistencyStatus` (via custom exporter scraping `clusterSet.status()`)	!= `OK`	Freeze writes, investigate

The last row does not have a built-in exporter. A common pattern is a small sidecar that periodically runs mysqlsh --js -e "print(JSON.stringify(dba.getClusterSet().status({extended:1})))" and exposes the parsed fields as a Prometheus textfile.

Step 7: Failover Procedures

7.1 Planned Switchover

For maintenance windows, DC migrations, and version upgrades. Zero data loss.

var cs = dba.getClusterSet()

// Promote DR
cs.setPrimaryCluster('MorphDrCluster')

// Switch back
cs.setPrimaryCluster('morpheus-cluster')

Realistic RTO is roughly 30 seconds to 2 minutes, dominated by Router metadata refresh and the in-flight transaction drain. With --conf-use-gr-notifications enabled, the Router side is typically under 10 seconds.

7.2 Emergency Failover (`forcePrimaryCluster`)

Use this when Master is completely unreachable.

Warning: forcePrimaryCluster() skips consistency checks. Transactions committed on Master but not yet replicated to DR will be lost. Expect a small RPO measured in seconds, not zero.

mysqlsh clusterAdmin@dr_node1:3306
var cs = dba.getClusterSet()
cs.forcePrimaryCluster('MorphDrCluster')

Critical: Fence the Old Master

Before or immediately after the force failover, make sure the old Master cannot accept writes when it comes back up. Otherwise the application may briefly hit the old Master through its Router and create divergent data. The preferred approach is to use the AdminAPI fencing commands (MySQL Shell 8.0.27 and later). They coordinate the fence across every node in the old Master cluster and instruct Router to stop routing to it, rather than relying on each DBA to toggle super_read_only by hand.

// Connect to ANY reachable node of the old (ex-primary) cluster
mysqlsh clusterAdmin@master_node1:3306
var cluster = dba.getCluster()

// Hard fence: blocks BOTH writes and reads via Router. Use when you want
// the old cluster completely out of traffic while you investigate.
cluster.fenceAllTraffic()

// Softer alternative: only blocks writes, keeps the cluster available for R/O.
// Useful if the old Master can safely keep serving reads from stale data.
// cluster.fenceWrites()

// After you've repaired the old Master and are ready to rejoin it as a REPLICA:
// cluster.unfenceWrites()   // only needed if you used fenceWrites()

What the AdminAPI fencing actually does, per the official Shell docs:

fenceAllTraffic() sets super_read_only = ON on every member, stops Group Replication on every member, and updates the metadata so Router stops routing any traffic to the cluster.
fenceWrites() sets super_read_only = ON on the primary and updates metadata so Router stops sending R/W traffic but still allows R/O. The cluster keeps replicating from the new primary through ClusterSet.
unfenceWrites() reverses fenceWrites(), so the cluster comes back online as a healthy replica.

If AdminAPI is unreachable (for example the old Master cluster is completely partitioned away), fall back in this order:

Remove the old Master's VIP, DNS entry, or load balancer registration so the application can't reach it.
Firewall-block ports 3306 and 6446 at the old site from the application subnet.
As a last resort, SET GLOBAL super_read_only = ON; manually on each old Master node. This is a per-node toggle, easy to forget on one host, and it does not update ClusterSet metadata.

Skipping fencing is how you end up with a split-brain when the old Master briefly reconnects to the network.

After Master Recovers, Rejoin It

// Run from the DR primary (now the ClusterSet primary)
var cs = dba.getClusterSet()
cs.rejoinCluster('morpheus-cluster')

7.3 The Most Common Rejoin Failure: Errant GTIDs on the Old Master

If the old Master committed transactions during the outage that never made it to DR, rejoinCluster() will fail with an errant-GTID error. In practice this is by far the most common post-failover pain point.

Diagnose:

-- On old Master primary
SELECT @@global.gtid_executed;
-- On new primary (DR)
SELECT @@global.gtid_executed;
-- Compute the diff:
SELECT GTID_SUBTRACT(
  '<old_master_executed>',
  '<dr_executed>'
) AS errant_gtids;

Any non-empty errant_gtids means the old Master has transactions DR never saw. You have three options, ordered from safest to fastest:

Extract and replay manually. Use mysqlbinlog on the old Master to dump only the errant GTIDs and then replay them against the new primary (DR). The slowest option, but no data loss.

   # Step 1: On the OLD Master, find the binlog files covering the outage window
   ls -lh /var/lib/mysql/binlog.*

   # Step 2: Dump only the errant GTIDs to a SQL file.
   #         --include-gtids takes the exact GTID set printed by GTID_SUBTRACT above.
   mysqlbinlog \
     --read-from-remote-server \
     --host=<old_master_ip> --port=3306 \
     --user=clusterAdmin --password \
     --include-gtids='<errant_gtids_from_GTID_SUBTRACT>' \
     --skip-gtids=false \
     binlog.000123 binlog.000124 binlog.000125 \
     > errant.sql

   # Step 3: Inspect errant.sql. Confirm the statements are safe to re-apply
   #         (no DROP/TRUNCATE you didn't expect, no migrations already run on DR).
   less errant.sql

   # Step 4: On the NEW primary (DR), replay as clusterAdmin
   mysql -h dr_node1 -u clusterAdmin -p < errant.sql

   # Step 5: Re-run the GTID diff from §7.3. errant_gtids should now be empty.

Why --skip-gtids=false: it preserves the original GTIDs so the new primary records them as executed. Otherwise a later rejoinCluster() of the old Master will still see them as errant.

Accept the loss and re-clone. Treat the old Master cluster as disposable, dissolve() it, and re-add it as a new replica cluster against the current primary (DR). The fastest option, but loses the errant transactions.
Forcibly reset and rejoin. On each old Master node:

   STOP GROUP_REPLICATION;
   RESET MASTER;                      -- MySQL 8.0.x; on 8.4+ use RESET BINARY LOGS AND GTIDS
   -- Then rejoinCluster() and let clone re-seed

Only safe if you have the original data preserved and audited elsewhere. This wipes the entire GTID history on the target node.

This is why backups are non-negotiable. Option 1 depends on binary log retention that covers the outage window. Set binlog_expire_logs_seconds accordingly. Seven days is a reasonable floor for most shops.

7.4 Router Behavior After Failover

State	Master Router	DR Router
Normal	Master Cluster → R/W (6446)	Master Cluster → R/W (WAN)
After failover	DR Cluster → R/W (auto)	DR Cluster → R/W (local)

No application connection string change is needed. Router reacts to ClusterSet topology changes via GR notifications and its metadata cache.

Architecture Summary

Component	Master Site Role	DR Site Role
Cluster type	Primary (R/W)	Replica (R/O until failover)
Intra-cluster replication	Group Replication (sync)	Group Replication (sync)
Cross-site replication	Source (donor)	Async + TLS from Master
Failover method	`setPrimaryCluster()` (planned)	`forcePrimaryCluster()` (emergency)
Router behavior	Always routes to the primary	Auto-updates after failover
Realistic RTO	30 s to 2 min (planned)	2 to 5 min (forced, includes fence + rejoin)
RPO	Zero (planned)	Near-zero; non-zero on force (typically 1 to 5 s)

Operational Tips and Best Practices

A switchover drill every month is the single highest-leverage habit. Failover should never be tested for the first time during a real disaster; a monthly setPrimaryCluster() round-trip proves the procedure still works and keeps on-call muscle memory sharp.

Log clusterSet.status({extended: 1}) every 60 seconds to a file. A simple cron-scraped JSON log often beats "enterprise" monitoring when you are trying to reconstruct what happened during an incident.

Put Routers behind a load balancer. A single Router IP is a single point of failure. At minimum, run two Routers per site behind HAProxy, Keepalived, or a cloud load balancer.

Back up independently of the cluster. ClusterSet protects against infrastructure failure, not logical failure. A DROP TABLE replicates. Use MySQL Enterprise Backup or Percona XtraBackup on an independent schedule, ideally to a third location that is neither Master nor DR.

Keep the rolling approach for MySQL version upgrades: upgrade DR first, observe for a week, switch over, upgrade the old Master, then switch back.

Watch cross-site RTT. Async replication tolerates latency well, but GTID apply throughput drops meaningfully above roughly 50 ms RTT for write-heavy workloads. Benchmark before assuming it will work for your traffic shape.

Rotate the internal mysql_innodb_cs_* replication account credentials on a schedule. AdminAPI creates them with strong random passwords but never rotates them on its own. Build a runbook for the rotation.

Keep binlog_expire_logs_seconds generous. Seven days is a reasonable floor for most production environments. It is the difference between manual recovery (option 1 in §7.3) and data loss (option 2).

Don't share credentials between clusterAdmin and routeruser. They have different privilege needs, and scoping them tightly limits blast radius.

Document your fencing procedure before you need it. When a real emergency failover happens, you do not want to be figuring out "how do we stop the old Master from accepting writes" on the fly.

Wrap-Up

InnoDB ClusterSet is a production-grade multi-site DR solution that can be introduced into a live environment without downtime, provided you respect three disciplines:

Rolling restart discipline, so you never break quorum.
Clean DR teardown followed by a TLS-hardened ClusterSet creation. Use dissolve() (or fallback dropMetadataSchema()) to clean DR, then create the ClusterSet with clusterSetReplicationSslMode: 'REQUIRED' (or VERIFY_IDENTITY on Shell 8.0.33+) and a scoped replicationAllowedHost. Never use % in production.
Failover realism. Plan for errant GTIDs, old-Master fencing, and backup-dependent recovery paths before you need them, not after.

Get those three right and your application keeps talking to the same Router address while your infrastructure is ready for the worst case: a full data center loss.

Automating MySQL InnoDB Cluster Deployment for HPE Morpheus Enterprise HA

Emre Baykal — Sun, 19 Apr 2026 17:02:48 +0000

Automating MySQL InnoDB Cluster Deployment for HPE Morpheus Enterprise HA Environments

A single Python script that turns a painful, multi-hour manual process into a guided 10-minute deployment.

🔗 GitHub Repository: emrbaykal/morpheus-innodb-cluster

👤 Author: Emre Baykal

📜 License: MIT

If you deploy HPE Morpheus Enterprise in a 3-Node HA configuration, the installer automatically clusters the Application Tier, OpenSearch, and RabbitMQ — but leaves the Transactional Database Tier (MySQL) entirely in your hands. You must stand up a resilient, production-grade MySQL InnoDB Cluster before you can even start the Morpheus HA installation.

The morpheus-innodb-cluster project solves this with a single interactive Python script backed by modular Ansible roles — it turns a multi-day manual task into a 10-minute guided wizard.

Introduction

If you've ever deployed HPE Morpheus Enterprise in a High Availability (HA) configuration, you know the drill: the application tier, messaging tier, and non-transactional database tier all install and cluster automatically across your three nodes — but the Transactional Database Tier is left entirely in your hands. That tier is MySQL, and for a production HA environment, it needs to be a properly configured, redundant, multi-node cluster.

As the HPE Morpheus Enterprise v8.1.0 HA Installation Overview clearly states:

"In this architecture, all tiers are deployed on three machines by HPE Morpheus Enterprise during the installation, with the exception of the Transactional Database Tier. This provides HA not just for the HPE Morpheus Enterprise Application Tier but all underlying tiers that support HPE Morpheus Enterprise. The Transactional Database Tier will remain external, either as a separate cluster or PaaS, following the supported services. An external MySQL cluster must still be set up outside of the HPE Morpheus Enterprise app nodes."

This means you are responsible for standing up a resilient, properly tuned MySQL cluster before you can even begin the Morpheus HA installation. There is no embedded option — Morpheus disables its internal MySQL (mysql['enable'] = false) and expects you to provide an external endpoint.

This is where the morpheus-innodb-cluster project comes in. It is a single interactive Python script, backed by modular Ansible roles, that automates the entire process of deploying a production-ready 3-node MySQL InnoDB Cluster on Ubuntu or RHEL-based systems.

In this post, I'll walk you through why this tool exists, what problems it solves, and how to use it from start to finish.

Understanding HPE Morpheus Enterprise HA Architecture

HPE Morpheus Enterprise is a unified hybrid cloud management platform that provides provisioning, orchestration, monitoring, and governance across private and public clouds. It can start as a simple single-machine instance, or it can be split into individual services per machine and configured in a high availability (HA) configuration.

According to the official HPE documentation (v8.1.0, February 2026), there are four primary tiers of services within the Morpheus appliance:

1. Application Tier

The stateless services layer — Nginx and Tomcat. These can be installed across all regions and placed behind a central load balancer or geo-based load balancer. Shared storage (NFS, Amazon S3, or OpenStack Swift) is required for deployment archives, virtual image catalogs, and backups.

2. Transactional Database Tier (MySQL) 🎯

This is the focus of this article. The HPE documentation states: "The Transactional Database tier consists of a MySQL compatible database. It is recommended that a lockable clustered configuration be used, such as a Galera cluster, which can also provide high availability." For the recommended 3-Node HA architecture, this tier must be external — it is not deployed by the Morpheus installer.

3. Non-Transactional Database Tier (OpenSearch)

Used for log aggregation, stats, metrics, and temporal data. Morpheus clusters OpenSearch automatically during installation — no manual setup required.

4. Messaging Tier (RabbitMQ)

An AMQP-based tier with STOMP protocol for agent communication. RabbitMQ needs at least 3 instances for HA. While RabbitMQ is installed automatically during Morpheus setup, it does require manual clustering afterward.

The Architecture Diagram

The recommended 3-Node HA deployment looks like this: three Morpheus application nodes sit behind a load balancer, each running embedded RabbitMQ and OpenSearch. On the side, a separate 3-node MySQL Database Cluster provides the transactional database tier, connected via port 3306. Shared storage connects to all application nodes.

HPE Morpheus Enterprise 3-Node HA Architecture — the external MySQL cluster on the right is exactly what this tool automates.

MySQL Requirements for Morpheus HA

The HPE 3-Node HA Install documentation specifies the following MySQL requirements:

MySQL version v8.0.x (minimum of v8.0.72)
MySQL cluster with at least 3 nodes for redundancy
Morpheus application nodes must have connectivity to the MySQL cluster

There is also an important note: "Morpheus does not create primary keys on all tables. If you use a clustering technology that requires primary keys, you will need to leverage the invisible primary key option in MySQL 8."

Once the MySQL cluster is up, you must create the Morpheus database and user before installing Morpheus itself:

-- Create the Morpheus database
CREATE DATABASE morpheus CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;

-- Create the Morpheus database user
CREATE USER 'morpheus'@'%' IDENTIFIED BY 'morpheusDbUserPassword';

-- Grant required permissions
GRANT ALL PRIVILEGES ON morpheus.* TO 'morpheus'@'%' WITH GRANT OPTION;
GRANT SELECT, PROCESS, SHOW DATABASES, RELOAD ON *.* TO 'morpheus'@'%';
FLUSH PRIVILEGES;

Then, in each Morpheus app node's /etc/morpheus/morpheus.rb, you configure:

mysql['enable'] = false
mysql['host'] = {'127.0.0.1' => 6446}   # MySQL Router local endpoint
mysql['morpheus_db'] = 'morpheus'
mysql['morpheus_db_user'] = 'morpheus'
mysql['morpheus_password'] = 'morpheusDbUserPassword'

Notice mysql['enable'] = false — this tells Morpheus to skip its embedded MySQL and use your external cluster instead. The host points to 127.0.0.1:6446, which is the MySQL Router read-write endpoint running locally on each app node.

The Pain of Manual MySQL InnoDB Cluster Setup

The HPE documentation tells you that you need an external MySQL cluster, but it doesn't deploy one for you. You're on your own. Here's what you're typically facing:

1. Repetitive Per-Node Configuration — OS tuning, kernel parameters, firewall rules for ports 3306, 33060, and 33061, NTP sync, locale settings, and MySQL installation on every node.

2. MySQL Installation Complexity — Repository management, correct stream/version selection, package locks, and systemd overrides. On RHEL, you also deal with AppStream module conflicts.

3. InnoDB-Specific Tuning — Enable GTID mode, enforce GTID consistency, set unique server_id, tune innodb_buffer_pool_size to RAM, configure binary log expiration, and bind addresses correctly.

4. Cluster Bootstrap Choreography — dba.configureInstance() on every node, dba.createCluster() on the primary, cluster.addInstance() for each secondary with the correct recovery method. Order matters.

5. Security Considerations — MySQL root passwords, cluster admin credentials, SSH keys, and sudo escalation — all of which need to be managed securely.

6. OS-Specific Differences — Ubuntu/Debian vs. RHEL differ in repository management, package names, services, paths, and security frameworks (AppArmor vs. SELinux).

Put it all together and you're looking at multiple hours — or days — of careful, error-prone work before you can even begin the Morpheus installation itself.

Introducing morpheus-innodb-cluster

The morpheus-innodb-cluster project eliminates all of this manual toil. It is a single Python script (innodb_cluster_setup.py) that orchestrates the entire deployment through an interactive 11-step wizard, backed by four modular Ansible roles that handle the actual configuration work.

Architecture at a Glance

You (on master node)
  └── innodb_cluster_setup.py  (Python orchestrator)
        ├── Step 1-9:  Interactive wizard (collect & validate)
        └── Step 10:   Ansible playbook execution
              ├── Role 01: OS Pre-configuration
              ├── Role 02: MySQL Installation
              ├── Role 03: InnoDB Cluster Pre-configuration
              └── Role 04: Cluster Creation (master only)
        └── Step 11:   Setup Report

The script is designed to run from the master node of the cluster. It installs its own dependencies (Ansible, sshpass, community.mysql collection), connects to all three nodes via SSH, and executes the Ansible playbook that does the heavy lifting. You don't need to pre-install anything other than Python 3.6+ and sudo access.

Step-by-Step Walkthrough

Let's walk through the entire deployment process, using screenshots from a real deployment on RHEL 9 nodes.

Getting Started

Clone the repository and run the script on the node you want to be the primary (master) node:

git clone https://github.com/emrbaykal/morpheus-innodb-cluster.git
cd morpheus-innodb-cluster
sudo python3 innodb_cluster_setup.py

Step 1/11 — Environment Setup

The script begins by detecting your operating system family and automatically installing all required prerequisites. On RHEL systems, it checks for the Ansible Automation Platform repository in subscription-manager; if it's not enabled, the script gracefully falls back to installing Ansible via pip3.

In the screenshot above, you can see the tool detecting RHEL 9, installing sshpass and python3-pip via the OS package manager, falling back to pip for Ansible (since the AAP repository wasn't enabled in subscription-manager), and installing the community.mysql Ansible collection. The entire environment is ready in seconds.

Step 2/11 — Cluster Configuration

This is the heart of the wizard. It collects all the information needed in five organized sections:

Section 1/5 — Cluster Nodes: You provide the hostname and IP address for each of the three cluster nodes. The first node is automatically designated as the master (primary), and the other two become secondaries.

Section 2/5 — SSH Connection: The script asks for the SSH user, key file (or password), privilege escalation method (sudo or dzdo), and sudo password.

Section 3/5 — MySQL Credentials: You set the MySQL root password and the InnoDB Cluster admin credentials. The cluster admin user (default: clusterAdmin) is the account that will manage the InnoDB Cluster.

Section 4/5 — Cluster Settings: You name your cluster (in our case, morpheus-cluster) and set a password for the MySQL Router user account (routeruser) that will be created for application connectivity.

Section 5/5 — System Settings: NTP server configuration for time synchronization across all nodes — critical for Group Replication to function correctly.

Configuration Summary

Before anything is written to disk or executed, the script presents a complete summary table of everything you've entered — cluster nodes with IPs, SSH connection details, MySQL configuration, and system settings. You review and confirm with y before proceeding.

This is a safety net. All passwords are masked, and the configuration is saved to cluster_config.json with mode 0600 so it can be reused on subsequent runs without re-entering everything.

Steps 3–4 — Inventory Generation & SSH Connectivity Test

The script generates an Ansible inventory file from your inputs and then tests SSH connectivity to every node. This catches authentication problems, firewall issues, or unreachable hosts before the deployment begins.

In our deployment, all three nodes — innodb-rhel-1 (192.168.42.100), innodb-rhel-2 (192.168.42.102), and innodb-rhel-3 (192.168.42.101) — came back as reachable. The inventory uses IP addresses throughout, making the deployment DNS-independent. Note the ansible_ssh_common_args='-o StrictHostKeyChecking=no' setting, which prevents SSH from blocking first-time connections.

Steps 5–7 — Pre-Flight Checks

The script performs three validation checks before deployment:

Step 5 — RHEL Repository Check: Verifies that rhel-9-for-x86_64-baseos-rpms and rhel-9-for-x86_64-appstream-rpms are enabled on all nodes.
Step 6 — Pre-Flight MySQL Package Check: Scans all nodes for pre-existing mysql-server and mysql-shell installations that could cause conflicts.
Step 7 — Internet Connectivity Check: Tests that all nodes can reach dev.mysql.com to download MySQL packages.

Step 8/11 — MySQL Version Selection

On RHEL systems, the script presents the available MySQL AppStream streams (8.0 and 8.4) and then lists the specific versions within your chosen stream. In our deployment, we selected MySQL 8.4.7 from the 8.4 LTS stream — well above the HPE minimum requirement of v8.0.72.

Step 9/11 — Deployment Confirmation

A final confirmation screen shows exactly what will be deployed: the cluster name (morpheus-cluster), admin user (clusterAdmin), and all three nodes with their roles. The master node is marked with a star.

Step 10/11 — Ansible Playbook Execution

Once you confirm, the Ansible playbook executes with real-time streaming output. You can watch every task as it runs across all three nodes. The playbook applies five roles in sequence:

pre_tasks — Populates /etc/hosts with cluster node entries and sets hostnames on all nodes
01-os-preconfigure — Deploys SSH banners, disables SELinux/AppArmor, configures firewall rules (ports 3306, 33060, 33061), sets locale, installs and configures NTP (chrony/systemd-timesyncd), tunes kernel parameters for database workloads, disables Transparent Huge Pages, and sets MySQL-specific system limits
02-mysql-install — Adds the MySQL repository, selects the AppStream stream/version, installs MySQL Server and MySQL Shell, configures systemd overrides with NUMA interleaving, and sets the root password
03-mysql-innodb-cluster — Creates the cluster admin user, removes anonymous users, drops the test database, writes InnoDB-optimized my.cnf configuration (with buffer pool auto-sized to 80% of RAM), enables GTID mode, and restarts MySQL
04-mysql-create-innodb-cluster — Runs on the master node only: configures instances via dba.configureInstance() in MySQL Shell, creates the cluster with dba.createCluster(), adds secondary nodes with clone-based recovery, verifies cluster status, and creates the routeruser account

In our deployment, the entire Ansible playbook completed in 4 minutes and 31 seconds with zero failures across all three nodes. ⚡

Step 11/11 — Setup Report

After completion, the script generates a comprehensive setup report that includes everything you need to verify and manage your new cluster.

The NODE STATUS section shows the Ansible PLAY RECAP for each node — in our deployment, the master node had 70 OK / 32 Changed tasks, while each secondary had 61 OK / 26 Changed, with zero unreachable or failed across the board.

The APPLIED ROLES section shows what each role did, and the NEXT STEPS section provides ready-to-use commands:

# Quick cluster status check
mysqlsh clusterAdmin@192.168.42.100 -- cluster status

# Interactive cluster management
mysqlsh clusterAdmin@192.168.42.100
var cluster = dba.getCluster();
cluster.status();

# Bootstrap MySQL Router (run on each Morpheus app node)
mysqlrouter --bootstrap routeruser@192.168.42.100:3306 --user=mysqlrouter

What Makes This Project Different

✅ Truly Idempotent

The entire workflow is safe to re-run. The script saves your configuration to cluster_config.json and offers to reuse it on subsequent runs. Ansible tasks check the current state before making changes. MySQL root password handling tries socket authentication first, then falls back to the existing password.

✅ OS-Aware Automation

The Ansible roles automatically detect ansible_os_family and execute the appropriate tasks for each distribution. Whether you're running Ubuntu 22.04, Ubuntu 24.04, RHEL 8, or RHEL 9, the same script handles everything.

✅ Production-Grade Tuning

This isn't just a "get MySQL running" script. It applies production-grade OS and database tuning:

Kernel parameters: TCP backlog, connection queue, TIME_WAIT reuse, swap minimization, dirty page ratios, file descriptor limits, and async I/O limits
MySQL limits: 65,535 open files, 65,535 processes, unlimited memory lock
NUMA interleaving: MySQL runs with numactl --interleave=all for optimal memory access on multi-socket servers
InnoDB buffer pool: Automatically sized to 80% of total RAM
GTID-based replication: Required for InnoDB Cluster and enabled automatically
Binary log management: Automatic purge after 7 days

✅ Security by Default

All generated configuration files are created with mode 0600. Passwords are never logged (Ansible no_log: true). Terminal input is masked for all password prompts. Temporary credential files in /tmp/ are cleaned up after cluster creation.

Connecting Morpheus to Your New Cluster

Once the InnoDB Cluster is running, you need to bridge it with your Morpheus application nodes. Here's the complete workflow:

1. Create the Morpheus Database

Log into your new cluster's primary node and create the database and user that Morpheus expects:

mysql -u root -p

CREATE DATABASE morpheus CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
CREATE USER 'morpheus'@'%' IDENTIFIED BY 'morpheusDbUserPassword';
GRANT ALL PRIVILEGES ON morpheus.* TO 'morpheus'@'%' WITH GRANT OPTION;
GRANT SELECT, PROCESS, SHOW DATABASES, RELOAD ON *.* TO 'morpheus'@'%';
FLUSH PRIVILEGES;

2. Bootstrap MySQL Router on Each App Node

Install MySQL Router on each Morpheus application node and bootstrap it against the cluster:

mysqlrouter --bootstrap routeruser@192.168.42.100:3306 --user=mysqlrouter
systemctl enable mysqlrouter && systemctl start mysqlrouter

MySQL Router will automatically discover all cluster members and create local read-write (port 6446) and read-only (port 6447) endpoints.

3. Configure Morpheus

Edit /etc/morpheus/morpheus.rb on each app node, pointing MySQL to the local router endpoint:

mysql['enable'] = false
mysql['host'] = {'127.0.0.1' => 6446}
mysql['morpheus_db'] = 'morpheus'
mysql['morpheus_db_user'] = 'morpheus'
mysql['morpheus_password'] = 'morpheusDbUserPassword'

Then reconfigure and proceed with the rest of the Morpheus HA installation:

morpheus-ctl reconfigure

With this setup, MySQL Router handles automatic failover. If the primary node goes down, Group Replication elects a new primary, and MySQL Router transparently redirects traffic — all without any Morpheus downtime.

Network Requirements

MySQL InnoDB Cluster Ports (between DB nodes)

Port	Protocol	Service
3306	TCP	MySQL Classic Protocol
33060	TCP	MySQL X Protocol
33061	TCP	Group Replication

Morpheus App Node to MySQL Cluster

Port	Protocol	Service
3306	TCP	MySQL connection (or via Router 6446/6447)

Additional Morpheus HA Ports (for reference)

Port	Protocol	Service
443	TCP	HTTPS (inbound from users/agents)
4369	TCP	RabbitMQ EPMD (inter-node discovery)
5671/5672	TCP	RabbitMQ (TLS/non-TLS)
9200	TCP	OpenSearch API
9300	TCP	OpenSearch inter-node
25672	TCP	RabbitMQ inter-node
61613/61614	TCP	STOMP (non-TLS/TLS)

PaaS Alternatives

The HPE documentation also lists supported PaaS offerings as alternatives to self-managed MySQL clusters:

Cloud	Database (MySQL)
AWS	Amazon Aurora
GCP	MySQL Instance
Azure	N/A
OCI	N/A
Alibaba	N/A

As you can see, PaaS support for MySQL is limited to AWS and GCP. For on-premises deployments, private cloud, or any other environment, a self-managed MySQL cluster is your only option — which is exactly what this tool provides.

Conclusion

Setting up a MySQL InnoDB Cluster for HPE Morpheus Enterprise HA shouldn't be a multi-day project requiring deep MySQL expertise. The morpheus-innodb-cluster tool reduces it to a single command and a 10-minute guided wizard. It handles OS detection, prerequisite installation, interactive configuration, pre-flight validation, Ansible-driven deployment, and post-deployment reporting — all in one cohesive workflow.

While HPE Morpheus Enterprise excels at automatically clustering OpenSearch and providing the framework for RabbitMQ clustering, the Transactional Database Tier remains the one piece that administrators must provision themselves. This tool fills that gap, giving you a consistent, repeatable, production-ready MySQL InnoDB Cluster every time.

Whether you're deploying Morpheus HA for the first time or rebuilding your database tier after a migration, you're three commands away:

git clone https://github.com/emrbaykal/morpheus-innodb-cluster.git
cd morpheus-innodb-cluster
sudo python3 innodb_cluster_setup.py

If you found this useful, give the GitHub repository a ⭐ and feel free to open issues or contribute.

References

HPE Morpheus Enterprise & VM Essentials SAML Integration with Keycloak: A Complete Technical Guide

Emre Baykal — Sat, 28 Mar 2026 17:26:07 +0000

1. Introduction

1.1 What is SAML 2.0?

SAML (Security Assertion Markup Language) 2.0 is an XML-based open standard for exchanging authentication and authorization data between two parties: an Identity Provider (IdP) that authenticates users, and a Service Provider (SP) that hosts the application. Instead of every application managing its own username/password database, SAML lets you delegate authentication to a central IdP. When a user logs in once at the IdP, they get access to all connected SPs without entering credentials again — this is Single Sign-On (SSO).

In practical terms: the user clicks "Login with SSO" on the application, gets redirected to the IdP login page, authenticates there, and is sent back to the application with a cryptographically signed XML document (the "SAML assertion") that proves who they are and what groups they belong to.

1.2 Why Keycloak?

There are several IdP options available (Okta, Azure AD, ADFS, Ping Identity, etc.), so why Keycloak?

Open-source and free — No per-user licensing costs, which matters at scale
Self-hosted — Full control over your identity infrastructure; no dependency on external SaaS providers
Protocol versatility — Supports SAML 2.0, OpenID Connect, and OAuth 2.0 in a single platform
LDAP/AD federation — Connects directly to Active Directory without migrating users
Kubernetes-native — Runs well as a containerized deployment with built-in clustering
CNCF project — Active community, regular releases, and long-term sustainability

1.3 What We Will Build

Enterprise environments demand centralized identity management. When you operate HPE Morpheus Enterprise as your cloud management platform, integrating it with a dedicated Identity Provider (IdP) via SAML 2.0 eliminates password sprawl and gives you single sign-on (SSO) across the entire infrastructure stack.

In this post, we walk through the entire journey: understanding how Morpheus handles SAML under the hood, deploying Keycloak on Kubernetes as your IdP, wiring the two together, and diagnosing issues when things do not go as planned. Every configuration value and YAML snippet comes from a real lab deployment, so you can replicate it in your own environment.

Lab environment: Kubernetes (single-node / Minikube compatible), Keycloak 26.x, Morpheus Enterprise 8.x, Active Directory for user federation.

2. Morpheus SAML Architecture

HPE Morpheus Enterprise supports SAML 2.0 as an Identity Source type within its Administration panel. Understanding how Morpheus participates in the SAML exchange is essential before configuring anything on the Keycloak side.

📸 IMAGE: Morpheus > Login Screen

📸 IMAGE: Morpheus > Forward Keycloak Login Screen

2.1 Morpheus as a SAML Service Provider (SP)

Morpheus acts exclusively as a SAML Service Provider. It does not function as an Identity Provider itself. When a user attempts to log in via SSO, Morpheus generates a SAML AuthnRequest, redirects the browser to the configured IdP, and then consumes the SAML Response (assertion) returned by the IdP.

2.2 Key SAML Endpoints in Morpheus

When you create a SAML Identity Source in Morpheus, the platform automatically generates two critical values:

Endpoint	Description
SP Entity ID	A unique identifier for Morpheus as a Service Provider. Auto-generated from the hostname, e.g. `https://morpheus-url/saml/<uniqueID>`
SP ACS URL	The callback URL where the IdP posts the SAML Response after authentication, e.g. `https://morpheus-url/externalLogin/callback/<uniqueID>`
Login Redirect URL	The IdP's SAML SSO endpoint where Morpheus sends the AuthnRequest
SAML Logout Redirect URL	The IdP's SAML SLO endpoint for single logout

Note: The SP Entity ID and ACS URL are generated only after you save the Identity Source for the first time. You must save first, then copy these values to configure the IdP.

2.3 SAML Request and Response Configuration

Morpheus provides granular control over how SAML requests are signed and responses are validated:

Setting	Options & Description
SAML Request	No Signature / Self Signed / Custom RSA Signature — Controls whether AuthnRequest messages are signed
SAML Response	Do Not Validate / Validate Assertion Signature — Controls signature validation on the IdP's assertion
POST Binding Mode	ON/OFF — Uses HTTP-POST binding instead of HTTP-Redirect
Includes SAML Request Parameter	Yes/No — Whether the SAML request is included in the redirect

2.4 Assertion Attribute Mappings

Morpheus maps SAML assertion attributes to internal user fields:

Morpheus Field	Expected SAML Attribute
Given Name	`firstName`
Surname	`lastName`
Email	`email` (or NameID)

2.5 Role Mapping Mechanism

Morpheus supports role-based access control through SAML group assertions:

Role Mapping Field	Description
Default Role	The role assigned to all authenticated users (e.g., Standard User)
Role Attribute Name	The SAML attribute containing group/role info (e.g., `groups`)
Required Role Attribute Value	A group name the user must belong to for authorization (e.g., `mspusers`)

📸 IMAGE: Morpheus > Identity Sources > Keycloak

3. Keycloak-SAML Ecosystem: How It Works

Keycloak is an open-source Identity and Access Management (IAM) solution maintained by the CNCF. It supports OpenID Connect, OAuth 2.0, and SAML 2.0 protocols natively. In our setup, Keycloak serves as the SAML Identity Provider (IdP) that authenticates users against an Active Directory backend via LDAP federation.

3.1 Core Keycloak Concepts

Concept	Description
Realm	A tenant-level isolation boundary. Each realm has its own users, clients, roles, and identity providers. Our realm: `morpheus-lab`
Client	An application that delegates authentication to Keycloak. Morpheus is registered as a SAML client
User Federation	Allows Keycloak to pull users from LDAP/Active Directory without duplicating credentials
Protocol Mappers	Transform user attributes and group memberships into SAML assertions
Roles & Groups	Realm-level and client-level roles. AD groups can be synced and mapped into assertions

3.2 SAML 2.0 Authentication Flow (SP-Initiated SSO)

The diagram below illustrates the SP-Initiated SAML SSO flow between Morpheus and Keycloak:

Step-by-step:

The user navigates to the Morpheus login page and clicks the SSO login button.
Morpheus generates a SAML AuthnRequest and redirects the user's browser to Keycloak's SAML endpoint: https://keycloak-server:30443/realms/morpheus-lab/protocol/saml
Keycloak presents the login form. The user enters their AD credentials.
Keycloak authenticates the user against the federated LDAP/AD backend.
On successful authentication, Keycloak constructs a SAML Response containing signed assertions with user attributes (firstName, lastName) and group memberships (groups).
Keycloak POSTs the SAML Response to the Morpheus ACS URL.
Morpheus validates the assertion signature, maps attributes and roles, creates or updates the user session, and grants access.

3.3 Authentication vs Authorization

Authentication (Who are you?)

Keycloak acts as the authentication broker, sitting between the application (Morpheus) and the identity store (Active Directory).
User Federation (LDAP) allows Keycloak to verify credentials against AD without storing passwords locally. Keycloak performs LDAP BIND operations to authenticate users.
MFA can be layered on top through Keycloak's authentication flow configuration.
Session Management: Once authenticated, Keycloak creates a session. Subsequent SAML requests within the session's lifetime do not require re-authentication (SSO behavior).

Authorization (What can you do?)

Keycloak syncs AD groups via the LDAP Group Mapper (mspusers, apparchitech, selfservice).
The SAML Group List Mapper serializes group memberships into the SAML assertion as a groups attribute.
Morpheus reads the groups attribute and maps it to internal roles:
- mspusers → authorized user (Required Role)
- apparchitech → Application Architect role
- selfservice → Self-Service User role
Users not in the required group are denied access even if authentication succeeds.

3.4 Single Logout (SLO) Flow

User clicks Logout → Morpheus sends LogoutRequest → Keycloak terminates session
→ Keycloak POSTs LogoutResponse to /login/auth → User lands on login page

Warning: The Logout Service POST Binding URL must be set to /login/auth (the login page), NOT the ACS callback URL. Morpheus's ACS handler cannot process LogoutResponse objects and throws a GroovyCastException.

4. Deploying Keycloak on Kubernetes

This section walks through deploying a production-grade Keycloak cluster on Kubernetes using a single YAML manifest.

The complete Kubernetes manifest used in this guide is available on GitHub:
keycloak.yaml on GitHub

4.1 Architecture Overview

The deployment stack:

Keycloak StatefulSet (2 replicas) with Infinispan clustering for HA
PostgreSQL Deployment with PersistentVolumeClaim (10Gi)
Kubernetes Secret for admin and database passwords
Self-signed TLS certificates generated by init containers
NodePort Service for external HTTPS access on port 30443
Headless Service for Infinispan/JGroups cluster discovery

4.2 Prerequisites

Infrastructure:

A running Kubernetes cluster or Minikube instance
kubectl configured and connected to your cluster
At least 4GB RAM and 2 CPU cores available for the Keycloak + PostgreSQL pods
A storage provisioner (default StorageClass or Rook-Ceph). For Minikube, enable it with:

minikube addons enable default-storageclass
minikube addons enable storage-provisioner

Minikube users: Start Minikube with sufficient resources:
minikube start --cpus=4 --memory=8192 --driver=docker

Network & DNS requirements:

The Kubernetes node IP must be reachable from the Morpheus server (for SAML redirects)
The Morpheus server hostname (e.g., morpheus-server) must be resolvable from both the user's browser and the Keycloak pods. If you're using a local domain, add entries to /etc/hosts on the machines or configure your internal DNS
Firewall rules: Ensure these ports are open between the components:

Source	Destination	Port	Protocol	Purpose
User Browser	Morpheus Server	443	HTTPS	Access Morpheus UI
User Browser	K8s Node	30443	HTTPS	Keycloak login page (SAML redirect)
Morpheus Server	K8s Node	30443	HTTPS	SAML backchannel (POST binding)
K8s Pod Network	AD Domain Controller	389	LDAP	User federation / authentication

Active Directory requirements:

A dedicated service account for Keycloak LDAP binding (e.g., svc-keycloak). This account needs read-only access to the Users container (CN=Users,DC=yourdomain,DC=local). It does not need Domain Admin privileges — basic "Read all user information" permission is sufficient
AD groups that will map to Morpheus roles (e.g., mspusers, apparchitech, selfservice) must exist and users must be members of the appropriate groups

4.3 Step 1: Prepare Secrets

The YAML uses a Kubernetes Secret to store sensitive credentials. The passwords are Base64-encoded:

# Encode your passwords
echo -n 'YourAdminPassword!' | base64
# Output: WW91*****UGFzc3***cmQh

echo -n 'YourDBPassword!' | base64
# Output: WW9************mQh

The Secret resource in the YAML:

apiVersion: v1
kind: Secret
metadata:
  name: keycloak-secret
  namespace: keycloak
type: Opaque
data:
  admin-password: <base64-encoded-admin-password>
  db-password: <base64-encoded-db-password>

Note: Never commit plain-text passwords to version control. Use a secrets manager (Vault, Sealed Secrets) in production.

4.4 Step 2: Namespace and Storage

apiVersion: v1
kind: Namespace
metadata:
  name: keycloak
  labels:
    app.kubernetes.io/part-of: sso-stack
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pvc
  namespace: keycloak
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 10Gi

4.5 Step 3: PostgreSQL Database

Keycloak requires an external database in production mode. Key points from our manifest:

containers:
  - name: postgres
    image: postgres:17
    env:
      - name: POSTGRES_USER
        value: keycloak
      - name: POSTGRES_PASSWORD
        valueFrom:
          secretKeyRef:
            name: keycloak-secret
            key: db-password
      - name: POSTGRES_DB
        value: keycloak
      - name: PGDATA
        value: /var/lib/postgresql/data/pgdata
    readinessProbe:
      exec:
        command: ["pg_isready", "-U", "keycloak"]
      initialDelaySeconds: 10
      periodSeconds: 5
    resources:
      requests:
        memory: 256Mi
        cpu: 100m
      limits:
        memory: 512Mi
        cpu: 500m

4.6 Step 4: Keycloak StatefulSet

The Keycloak deployment uses a StatefulSet with 2 replicas for high availability.

Init Containers:

wait-for-postgres: Polls PostgreSQL port 5432 before Keycloak starts
generate-tls-cert: Generates a self-signed TLS certificate (10 years validity)

initContainers:
  - name: generate-tls-cert
    image: alpine:3.19
    command: ["sh", "-c"]
    args:
      - |
        apk add --no-cache openssl
        openssl req -x509 -nodes -days 3650 \
          -newkey rsa:2048 \
          -keyout /certs/tls.key \
          -out /certs/tls.crt \
          -subj "/CN=keycloak-morpheus-lab/O=Lab/C=TR"

Key Environment Variables:

Variable	Purpose
`KC_BOOTSTRAP_ADMIN_USERNAME`	Initial admin username (`admin`)
`KC_BOOTSTRAP_ADMIN_PASSWORD`	Admin password from Secret
`KC_DB` / `KC_DB_URL_HOST`	Database type and host (`postgres`)
`KC_HTTPS_CERTIFICATE_FILE`	Path to TLS certificate
`KC_HTTPS_CERTIFICATE_KEY_FILE`	Path to TLS key
`KC_CACHE`	Clustering mode (`ispn` = Infinispan)
`KC_HOSTNAME_STRICT`	Disabled for NodePort/self-signed setups
`KC_FEATURES`	Additional features (`token-exchange`)
`KC_HEALTH_ENABLED`	Enables `/health/*` endpoints for probes

Health Probes:

startupProbe: /health/started — 1s interval, 600 retries (10-minute startup window)
readinessProbe: /health/ready — Every 10s, 3 failures to mark unready
livenessProbe: /health/live — Every 10s, 3 failures to restart pod

4.7 Step 5: Services

Service	Type & Purpose
`keycloak` (ClusterIP)	Internal access: ports 8080 (HTTP) and 8443 (HTTPS)
`keycloak-discovery` (Headless)	JGroups cluster member discovery on port 7800
`keycloak-nodeport` (NodePort)	External access: port 30080 (HTTP) and 30443 (HTTPS)

4.8 Step 6: Deploy

# Apply the complete manifest
kubectl apply -f keycloak.yaml

# Watch the rollout
kubectl rollout status statefulset/keycloak -n keycloak --timeout=10m

# Verify all pods are running
kubectl get pods -n keycloak

# Access the admin console
# https://<node-ip>:30443/admin

Tip: For Minikube, use minikube ip to get the node IP. For a multi-node cluster, any node's IP will work with NodePort.

📸 IMAGE: Terminal output of kubectl get pods -n keycloak showing all pods running

5. Keycloak-Morpheus SAML Integration Guide

With Keycloak deployed and running, we can now configure the SAML integration. The configuration involves both the Keycloak side (IdP) and the Morpheus side (SP), and the order matters.

Important — Configuration Order:
There is a chicken-and-egg situation here. Keycloak needs the SP Entity ID and ACS URL to create the SAML client, but these values are auto-generated by Morpheus only after you save an Identity Source. The correct order is:

Create the Keycloak Realm and LDAP Federation first (Steps 1-2)

Create a preliminary Identity Source in Morpheus to obtain the SP Entity ID and ACS URL (Step 3)

Use those values to create the SAML Client in Keycloak (Steps 4-7)

Come back to Morpheus and complete the Identity Source configuration (Step 8)

5.1 Step 1: Create the Keycloak Realm

Log in to Keycloak Admin Console at https://keycloak-server:30443/admin
Click the realm dropdown (top-left) and select "Create Realm"
Set Realm Name to: morpheus-lab
Click Create

📸 IMAGE: Keycloak Admin Console > Manage Realm > Create Realm

5.2 Step 2: Configure LDAP User Federation

Navigate to morpheus-lab > User Federation > Add New provider > LDAP and configure:

Setting	Value
Name	`LAB AD`
Vendor	`Active Directory`
Connection URL	`ldap://active-directory:389`
Bind Type	`simple`
Bind DN	`CN=svc-keycloak,CN=Users,DC=domain,DC=domain`
Users DN	`CN=Users,DC=domain,DC=domain`
Username LDAP Attribute	`sAMAccountName`
Edit Mode	`READ_ONLY`
Import Users	`ON`

Add Group Mapper (Mappers > Add mapper):

Setting	Value
Mapper Type	`group-ldap-mapper`
Groups DN	`CN=Users,DC=domain,DC=domian`
Group Name LDAP Attribute	`cn`
Membership LDAP Attribute	`member`
Mode	`READ_ONLY`
User Roles Retrieve Strategy	`LOAD_GROUPS_BY_MEMBER_ATTRIBUTE`

Note: After saving, click 'Sync all users' and 'Sync LDAP groups to Keycloak' to import users and groups from Active Directory.

📸 IMAGE: Keycloak > User Federation > LAB AD settings

5.3 Step 3: Create Preliminary Identity Source in Morpheus (Get SP Entity ID)

Before creating the SAML client in Keycloak, we need to obtain the SP Entity ID and ACS URL from Morpheus. These values are auto-generated and unique to your Morpheus instance.

Log in to Morpheus at https://morpheus-server
Navigate to Administration > Identity Sources
Click + Add Identity Source
Set Type to SAML SSO
Set Name to Keycloak-SSO
For now, enter any placeholder URL in Login Redirect URL (e.g., https://placeholder.local) — we will update this later
Click Save Changes

After saving, Morpheus generates and displays two critical values in the Identity Source list:

SP Entity ID — e.g., https://morpheus-server/saml/N3h***B***O
SP ACS URL — e.g., https://morpheus-server/externalLogin/callback/N***3***O

Copy both of these values! You will need them in the next step to configure the SAML client in Keycloak. The N3****BO part is a unique identifier generated by your Morpheus instance — yours will be different.

📸 IMAGE: Morpheus > Identity Sources list showing the auto-generated SP Entity ID and ACS URL

5.4 Step 4: Create the SAML Client in Keycloak

Now go back to the Keycloak Admin Console and create the SAML client using the values from the previous step:

Navigate to morpheus-lab > Clients > Create Client
Set Client Type to SAML
Set Client ID to the SP Entity ID you copied from Morpheus: https://morpheus-server/saml/N3h**K**5**
Set Name to: Morpheus Enterprise

Access Settings:

Setting	Value
Root URL	`https://morpheus-server`
Valid Redirect URIs (ACS URL)	`https://morpheus-server/externalLogin/callback/N3hK5**`
Master SAML Processing URL	`https://morpheus-server/externalLogin/callback/N3hK5**`
IDP-Initiated SSO URL Name	`morpheus`

SAML Capabilities:

Setting	Value
Name ID Format	`username`
Force POST Binding	`ON`
Include AuthnStatement	`ON`

Signature & Encryption:

Setting	Value
Sign Documents	`ON`
Sign Assertions	`ON`
Signature Algorithm	`RSA_SHA256`
Client Signature Required	`OFF` (CRITICAL!)
Encrypt Assertions	`OFF`

Warning: Client Signature Required must be OFF. Morpheus signs requests with a self-signed certificate that does not match the certificate registered in Keycloak. If this is ON, every SAML request from Morpheus will be rejected.

📸 IMAGE: Keycloak > Clients > Morpheus Enterprise > Settings

5.5 Step 5: Configure Logout (Advanced Settings)

Navigate to Advanced tab > Fine Grain SAML Endpoint Configuration:

Setting	Value
Logout Service POST Binding URL	`https://morpheus-server/login/auth`

This is crucial! Setting this URL to /login/auth prevents the GroovyCastException bug. The Morpheus ACS callback handler cannot process SAML LogoutResponse objects.

5.6 Step 6: Configure SAML Mappers

Navigate to Client Scopes > dedicated > Mappers and add:

Mapper 1: Groups (Group List)

Setting	Value
Mapper Type	`Group list`
SAML Attribute Name	`groups`
SAML Attribute NameFormat	`Basic`
Single Group Attribute	`OFF`
Full Group Path	`OFF`

Mapper 2: firstName (User Attribute)

Setting	Value
Mapper Type	`User Attribute`
User Attribute	`firstName`
SAML Attribute Name	`firstName`

Mapper 3: lastName (User Attribute)

Setting	Value
Mapper Type	`User Attribute`
User Attribute	`lastName`
SAML Attribute Name	`lastName`

📸 IMAGE: Keycloak > Client Scopes > dedicated > Mappers list

5.7 Step 7: Copy the Realm Certificate

Navigate to morpheus-lab > Realm Settings > Keys
Find the RS256 key row and click the Certificate button
Copy the entire certificate string

📸 IMAGE: Keycloak > Realm Settings > Keys > RS256 certificate

5.8 Step 8: Complete the Morpheus Identity Source Configuration

Now go back to the preliminary Identity Source you created in Step 3 and update it with the real values. Navigate to Administration > Identity Sources > Keycloak-SSO > Edit:

Setting	Value
Login Redirect URL	`https://keycloak-server:30443/realms/morpheus-lab/protocol/saml`
SAML Logout Redirect URL	`https://keycloak-server:30443/realms/morpheus-lab/protocol/saml`
Includes SAML Request Parameter	`Yes`
POST Binding Mode	`ON`
SAML Request	`Self Signed`
SAML Response	`Validate Assertion Signature`
SAML Response Public Key	(paste RS256 certificate from Keycloak)

Assertion Attribute Mappings:

Morpheus Field	Value
Given Name Attribute Name	`firstName`
Surname Attribute Name	`lastName`

Role Mappings:

Morpheus Field	Value
Default Role	`Standard User`
Role Attribute Name	`groups`
Required Role Attribute Value	`mspusers`
Application Architect Role	`apparchitech`
Self Service User Role	`selfservice`

Click Save Changes.

📸 IMAGE: Morpheus > Identity Sources > Keycloak-SSO configuration

5.9 Step 9: Test the Integration

Open a new browser / incognito window
Navigate to https://morpheus-server
Click the SSO login option (Keycloak-SSO should appear)
You will be redirected to Keycloak's login page
Enter an AD username (e.g., emre.baykal) and password

6. On success, you will be redirected back to Morpheus and logged in

6. Troubleshooting SAML Integration

SAML integrations can fail silently or produce cryptic errors. This section covers the most common issues.

6.1 Diagnostic Tools

Tool	Usage
SAML Tracer (Browser Extension)	Captures SAML requests/responses in real-time. Available for Firefox and Chrome
Keycloak Events	Enable in Realm Settings > Events. Shows auth attempts and errors
Keycloak Logs	`kubectl logs statefulset/keycloak -n keycloak -f`
Morpheus Logs	`/var/log/morpheus/morpheus-ui/current`
Base64 Decoder	`echo '' \

6.2 Common Issues and Solutions

Issue 1: 'Invalid Requester' Error


Symptom	Keycloak returns 'Invalid requester' or 'Client not found'
Cause	SP Entity ID in Morpheus doesn't match Client ID in Keycloak
Solution	Copy the exact SP Entity ID from Morpheus Identity Source and use it as Client ID in Keycloak

Issue 2: Signature Validation Failure


Symptom	'Signature validation failed' or 'Invalid signature'
Cause	SAML Response Public Key in Morpheus doesn't match Keycloak's RS256 certificate
Solution	Copy fresh certificate from Keycloak > Realm Settings > Keys > RS256 > Certificate. Must be updated after every Keycloak redeployment

Issue 3: User Authenticated but Access Denied


Symptom	User authenticates at Keycloak but gets 'Access Denied' in Morpheus
Cause	User not in the required group, or Group List mapper missing
Solution	1) Verify user is in {% raw %}`mspusers` AD group. 2) Check Group List mapper in client scopes. 3) Use SAML Tracer to verify `groups` attribute in assertion

Issue 4: GroovyCastException on Logout


Symptom	500 error on logout with `GroovyCastException` in logs
Cause	LogoutResponse sent to ACS URL instead of login page
Solution	Set Logout Service POST Binding URL to `https://morpheus-server/login/auth` in Keycloak Advanced settings

Issue 5: Login Loop / Redirect Loop


Symptom	Browser keeps redirecting between Morpheus and Keycloak
Cause	Mixed HTTP/HTTPS, clock skew, or invalid ACS URL
Solution	1) Ensure both use HTTPS. 2) Verify NTP sync (SAML assertions are time-sensitive). 3) Check Valid Redirect URIs matches exactly

Issue 6: Attributes Not Mapped


Symptom	firstName/lastName are empty, roles not assigned
Cause	SAML mappers missing or attribute names don't match
Solution	1) Add User Attribute mappers for `firstName` and `lastName`. 2) Names are case-sensitive. 3) Use SAML Tracer to verify attributes in assertion XML

Issue 7: LDAP Users Not Appearing


Symptom	No users appear after LDAP federation setup
Cause	Wrong Bind DN, Users DN, or network connectivity
Solution	1) Test connectivity from pod. 2) Verify Bind DN has read access. 3) Click 'Sync all users'. 4) Check Keycloak logs for LDAP errors

6.3 SAML Assertion Debugging Checklist

When debugging, use SAML Tracer to capture the assertion and verify:

NameID: Present and in expected format (username)?
Issuer: Matches the Keycloak realm URL?
AudienceRestriction: Audience matches Morpheus SP Entity ID?
Conditions/NotBefore/NotOnOrAfter: Valid timestamps? Check for clock skew
AuthnStatement: Present? (Required by Morpheus)
AttributeStatement: firstName, lastName, groups attributes present with correct values?
Signature: Assertion signed? Certificate matches?

6.4 Useful Debug Commands

# Check Keycloak pod logs
kubectl logs -f statefulset/keycloak -n keycloak

# Decode a SAML Response from browser
echo '<base64-saml-response>' | base64 -d | xmllint --format -

# Test LDAP connectivity from inside the cluster
kubectl exec -it keycloak-0 -n keycloak -- \
  sh -c 'nc -zv domain-controller 389'

# Check Keycloak health
kubectl exec -it keycloak-0 -n keycloak -- \
  curl -sk https://localhost:8443/health/ready

7. Conclusion

Integrating Morpheus Enterprise with Keycloak via SAML provides a robust, centralized authentication solution for enterprise cloud management. Key takeaways:

Morpheus acts as a SAML SP; Keycloak acts as the SAML IdP with AD backend
The Kubernetes deployment uses a StatefulSet with Infinispan clustering for HA
Self-signed TLS certificates are generated by init containers — no external cert-manager required for lab environments
Client Signature Required must be OFF in Keycloak for Morpheus compatibility
The Logout Service POST Binding URL must point to /login/auth to avoid the GroovyCastException bug
Always update the SAML Response Public Key in Morpheus after redeploying Keycloak

Have questions or ran into a different issue? Drop a comment below!

Written by Emre Baykal — March 2026