Implementing a RHEL HA Add-On Cluster with SBD fencing for SAP Solutions on IBM Power10 Servers

Introduction

Red Hat Enterprise Linux (RHEL) for SAP provides a range of solutions for building highly available SAP environments for SAP applications. See Red Hat HA Solutions for SAP HANA, S/4HANA and NetWeaver based SAP Applications for an overview of available scenarios.

This blog describes a sample setup of a RHEL HA Add-On cluster on IBM Power10 servers that can be used as a basis for implementing high availability scenarios for SAP applications. The cluster uses SBD (STONITH Block Device) for node fencing.

The cluster uses IBM Power10 server partitions (LPARs) as cluster nodes. The information describes installing the high availability packages on each cluster node, setting up the cluster, and configuring the fencing devices.

Target audience: Architects and specialists who are planning or implementing a high availability deployment of SAP applications on IBM Power10 servers.

Conventions for shell examples later in this blog: All commands are executed as root and start with the # sign as the root prompt. Some commands use the continuation character to break the command into two or more lines. The output of some commands is omitted and replaced with three dots (…) instead.

Cluster node fencing on IBM Power servers

STONITH is an acronym for Shoot The Other Node In The Head and protects your data from corruption in a split-brain situation. A robust fencing mechanism is essential for a cluster to prevent split-brain problems. You must enable STONITH (fencing) for a RHEL HA Add-On production cluster.

SBD implements a fencing mechanism through exchanging messages via shared disk devices. SBD-based fencing requires a suitable watchdog timer device. The IBM PowerVM hypervisor implements such a watchdog timer service for Power10 processor-based servers with firmware level MH1030 or higher.

For more information about SBD-based fencing, see the Red Hat Knowledge Base article Design Guidance for RHEL High Availability Clusters – sbd Considerations.

The fence agents fence_sbd and fence_lpar are the two supported fencing agents for implementing STONITH devices on IBM Power servers. fence_lpar provides an alternative fencing mechanism that uses the HMC (Hardware Management Console) and is also supported on older IBM Power servers. For more information, see the Red Hat Knowledge Base article How can I configure Power fencing for the IBM Power platform using an HMC in a RHEL High Availability cluster?

It is recommended to use fence_sbd if possible, as it simplifies the setup and also the operational behaviour.

Prerequisites

The prerequisites for using SBD-based fencing in LPARs running on IBM Power servers are:

IBM Power10 servers with firmware level MH1030 or higher (see Power10 System Firmware Fix History)Cluster nodes must run in Power10 compatibility modeRed Hat Enterprise Linux version 9.2 or higherCluster nodes installed and configured according to the requirements of the SAP applicationOne or more shared storage devices exclusive to the STONITH configuration

Checking the processor type and compatibility mode

On both nodes, run the following command to verify that the processor compatibility mode is set to Power10. The AT_BASE_PLATFORM field shows the actual processor type and the AT_PLATFORM field shows the processor compatibility mode. The processor compatibility mode must be set to Power10. Otherwise, the hypervisor watchdog won’t be available.

# LD_SHOW_AUXV=1 /bin/true | grep _PLATFORM
AT_PLATFORM: power10
AT_BASE_PLATFORM: power10

Checking the Power10 hypervisor watchdog

On both nodes, run the following commands to verify that the Power10 hypervisor watchdog driver is loaded and the watchdog device is available. If one or both commands fail, make sure that the servers and LPARs meet the requirements documented above.

# lsmod | grep pseries_wdt
pseries_wdt 262144 0

# wdctl /dev/watchdog
Device: /dev/watchdog
Identity: pseries-wdt [version 0]
Timeout: 60 seconds
Pre-timeout: 0 seconds
FLAG DESCRIPTION STATUS BOOT-STATUS
KEEPALIVEPING Keep alive ping reply 1 0
MAGICCLOSE Supports magic close char 0 0
PRETIMEOUT Pretimeout (in seconds) 0 0
SETTIMEOUT Set timeout (in seconds) 0 0

Installing and configuring a RHEL HA Add-On cluster

Use the following steps to set up a two-node cluster on IBM Power10 servers. You must perform some steps on both nodes and some steps on either cluster node.

Installing the RHEL HA Add-On software

Install the software packages that are required for the implementation of a cluster.

Checking the RHEL HA repository

On both nodes, use the dnf repolist command to check the configured repositories. If the high-availability repository is missing, use the subscription-manager command to enable it.

# dnf repolist
Updating Subscription Management repositories.
repo id repo name
rhel-9-for-ppc64le-appstream-e4s-rpms Red Hat Enterprise Linux 9 for Power, little endian – AppStream – Update Services for SAP Solutions (RPMs)
rhel-9-for-ppc64le-baseos-e4s-rpms Red Hat Enterprise Linux 9 for Power, little endian – BaseOS – Update Services for SAP Solutions (RPMs)
rhel-9-for-ppc64le-highavailability-e4s-rpms Red Hat Enterprise Linux 9 for Power, little endian – High Availability – Update Services for SAP Solutions (RPMs)
rhel-9-for-ppc64le-sap-solutions-e4s-rpms Red Hat Enterprise Linux 9 for Power, little endian – SAP Solutions – Update Services for SAP Solutions (RPMs)

# subscription-manager repos –enable=”rhel-9-for-ppc64le-highavailability-e4s-rpms”
Repository ‘rhel-9-for-ppc64le-highavailability-e4s-rpms’ is enabled for this system.# dnf clean all
Updating Subscription Management repositories.
46 files removed

Installing the RHEL HA Add-On software packages

On both nodes, run the following command to install the necessary software packages.

# dnf -y install pcs pacemaker sbd fence-agents-sbd
…

Configuring a RHEL HA Add-On cluster

Configuring the firewall services

On both nodes, run the following commands to add the high availability service to the RHEL firewall if firewalld.service is installed and enabled.

# firewall-cmd –permanent –add-service=high-availability
success

# firewall-cmd –reload
success

Starting the PCS daemon

Start the PCS daemon that is used for controlling and configuring RHEL HA Add-On clusters through PCS. On both nodes, run the following commands to enable the service and then verify that it is running

# systemctl enable –now pcsd.service
Created symlink /etc/systemd/system/multi-user.target.wants/pcsd.service → /usr/lib/systemd/system/pcsd.service.

# systemctl status pcsd.service
● pcsd.service – PCS GUI and remote configuration interface
Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; preset: disabled)
Active: active (running) since Mon 2024-08-05 13:05:32 CEST; 17s ago
Docs: man:pcsd(8)
man:pcs(8)
Main PID: 42235 (pcsd)
Tasks: 29 (limit: 312708)
Memory: 248.3M
CPU: 1.963s
CGroup: /system.slice/pcsd.service
├─42235 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42238 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42248 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42249 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42251 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42255 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42258 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42263 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42267 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42271 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42277 /usr/bin/python3 -Es /usr/sbin/pcsd
└─42282 /usr/bin/python3 -Es /usr/sbin/pcsd

Populating entries for each node in the hosts file

On both nodes, add the IP addresses and host names of both hosts to the /etc/hosts file. For more information, see Setting up /etc/hosts files on RHEL cluster nodes.

Setting a password for the hacluster user ID

On both nodes, run the passwd command and enter a password for the hacluster user ID.

# passwd hacluster
Changing password for user hacluster.
New password:
Retype new password:
passwd: all authentication tokens updated successfully.

Authenticating the cluster nodes

Use the following command to authenticate user hacluster to the PCS daemon on the nodes in the cluster. The command prompts you for the password you set in the previous step. On any node, run the following command.

# pcs host auth hdbna1 hdbnb1 -u hacluster
Password:
hdbnb1: Authorized
hdbna1: Authorized

Configuring and starting the cluster nodes

Configure the cluster configuration file and synchronise the configuration to the specified nodes. The –start option also starts the cluster service on the nodes. On any node, run the following commands to set up the cluster and then check the status of the new cluster.

# pcs cluster setup SAP_CLUSTER –start hdbna1 hdbnb1
No addresses specified for host ‘hdbna1’, using ‘hdbna1’
No addresses specified for host ‘hdbnb1’, using ‘hdbnb1’
Destroying cluster on hosts: ‘hdbna1’, ‘hdbnb1’…
hdbna1: Successfully destroyed cluster
hdbnb1: Successfully destroyed cluster
Requesting remove ‘pcsd settings’ from ‘hdbna1’, ‘hdbnb1’
hdbna1: successful removal of the file ‘pcsd settings’
hdbnb1: successful removal of the file ‘pcsd settings’
Sending ‘corosync authkey’, ‘pacemaker authkey’ to ‘hdbna1’, ‘hdbnb1’
hdbna1: successful distribution of the file ‘corosync authkey’
hdbna1: successful distribution of the file ‘pacemaker authkey’
hdbnb1: successful distribution of the file ‘corosync authkey’
hdbnb1: successful distribution of the file ‘pacemaker authkey’
Sending ‘corosync.conf’ to ‘hdbna1’, ‘hdbnb1’
hdbna1: successful distribution of the file ‘corosync.conf’
hdbnb1: successful distribution of the file ‘corosync.conf’
Cluster has been successfully set up.
Starting cluster on hosts: ‘hdbna1’, ‘hdbnb1’…

# pcs cluster status
Cluster Status:
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 13:31:51 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbna1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 13:31:51 2024
* Last change: Mon Aug 5 13:30:49 2024 by hacluster via crmd on hdbna1
* 2 nodes configured
* 0 resource instances configured
Node List:
* Online: [ hdbna1 hdbnb1 ]

PCSD Status:
hdbna1: Online
hdbnb1: Online

Configuring the cluster fencing device

Use the following information to create the mandatory fencing device.

Validating the watchdog timer device to use with sbd

You can use the following command to verify that the hypervisor watchdog works properly. The command causes the node to reset after a short countdown.

# sbd -w /dev/watchdog test-watchdog

WARNING: This operation is expected to force-reboot this system
without following any shutdown procedures.

Proceed? [NO/Proceed] Proceed

Initializing /dev/watchdog with a reset countdown of 5 seconds …

NOTICE: The watchdog device is expected to reset the system
in 5 seconds. If system remains active beyond that time,
watchdog may not be functional.

Reset countdown … 5 seconds
Reset countdown … 4 seconds
Reset countdown … 3 seconds
Reset countdown … 2 seconds
System expected to reset any moment …

For more information, see the Red Hat support article Administrative Procedures for RHEL High Availability Clusters – Validating a Watchdog Timer Device (WDT) to Use with sbd.

Getting the WWID of the STONITH SBD devices

To configure SBD STONITH, you need one or more shared storage devices. Use pvs –all or multipath -l and note the WWIDs of the shared disk devices that are assigned to both LPARs.

The following example shows a setup with a single SBD device.

Creating an SBD device

On any node, run the following commands to set up and then enable the SBD device.

# pcs stonith sbd device setup
device=/dev/mapper/36005076812808157b80000000000124b
WARNING: All current content on device(s) ‘/dev/mapper/36005076812808157b80000000000124b’ will be overwritten
Type ‘yes’ or ‘y’ to proceed, anything else to cancel: y
Initializing device ‘/dev/mapper/36005076812808157b80000000000124b’…
Device initialized successfully

# pcs stonith sbd enable
device=/dev/mapper/36005076812808157b80000000000124b
SBD_WATCHDOG_TIMEOUT=10 SBD_STARTMODE=clean
Running SBD pre-enabling checks…
hdbnb1: SBD pre-enabling checks done
hdbna1: SBD pre-enabling checks done
Distributing SBD config…
hdbna1: SBD config saved
hdbnb1: SBD config saved
Enabling sbd…
hdbna1: sbd enabled
hdbnb1: sbd enabled
Warning: Cluster restart is required in order to apply these changes.

Checking the SBD configuration

Check the contents of the SBD configuration file. Since the configuration file was generated by pcs, the contents should be the same on both nodes.

# cat /etc/sysconfig/sbd
# This file has been generated by pcs.
SBD_DELAY_START=no
SBD_DEVICE=”/dev/mapper/36005076812808157b80000000000124b”
SBD_OPTS=”-n hdbna1″
SBD_PACEMAKER=yes
SBD_STARTMODE=clean
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=10

The following commands show some more details about the available watchdog devices.

# pcs stonith sbd watchdog list
Available watchdog(s):
/dev/watchdog
/dev/watchdog0

# sbd query-watchdog

Discovered 2 watchdog devices:

[1] /dev/watchdog
Identity: pseries-wdt
Driver: pseries-wdt

Testing the watchdog with a stonith operation

On both nodes, run the following command to verify that the stonith operation forces the node to reboot.

# pcs stonith sbd watchdog test /dev/watchdog
WARNING: This operation is expected to force-reboot this system without following any shutdown procedures
Type ‘yes’ or ‘y’ to proceed, anything else to cancel: yes
System will reset shortly

Creating the STONITH device

The following command shows the device-specific options for the fence_sbd fencing agent. Alternatively, you can check its man pages with man fence_sbd.

# pcs stonith describe fence_sbd
…

On any node, start the cluster and then run the pcs stonith create command to create the STONITH device.

# pcs cluster start –all
hdbna1: Starting Cluster…
hdbnb1: Starting Cluster…

# pcs stonith create res_fence_sbd fence_sbd
devices=/dev/mapper/36005076812808157b80000000000124b

Verifying the cluster configuration

Use the following commands to display the STONITH configuration and the status of the SBD device.

# pcs stonith config
Resource: res_fence_sbd (class=stonith type=fence_sbd)
Attributes: res_fence_sbd-instance_attributes
devices=/dev/mapper/36005076812808157b80000000000124b
Operations:
monitor: res_fence_sbd-monitor-interval-60s
interval=60s

# pcs stonith sbd config
SBD_DELAY_START=no
SBD_STARTMODE=clean
SBD_WATCHDOG_TIMEOUT=10

Watchdogs:
hdbna1: /dev/watchdog
hdbnb1: /dev/watchdog

Devices:
hdbna1: “/dev/mapper/36005076812808157b80000000000124b”
hdbnb1: “/dev/mapper/36005076812808157b80000000000124b”# pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbna1: YES | YES | YES
hdbnb1: YES | YES | YES

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 clear
1 hdbnb1 clear# pcs stonith sbd status –full
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbna1: YES | YES | YES
hdbnb1: YES | YES | YES

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 clear
1 hdbnb1 clear

Testing the fencing of the nodes

To test the STONITH configuration, you must manually fence the nodes. On the first node, run the following commands to fence the second node and then view the cluster status.

# pcs stonith fence hdbnb1
Node: hdbnb1 fenced

# pcs status
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 15:07:41 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbna1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 15:07:41 2024
* Last change: Mon Aug 5 14:59:59 2024 by root via cibadmin on hdbna1
* 2 nodes configured
* 1 resource instance configured

Node List:
* Online: [ hdbna1 ]
* OFFLINE: [ hdbnb1 ]

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbna1

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled

The command has resulted in a reboot of the second node.
After the reboot, log in to the second node. Then, you must clear the SBD device before starting the cluster.

# pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbna1: YES | YES | YES
hdbnb1: YES | YES | NO

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 clear
1 hdbnb1 reset hdbna1.isicc.de.ibm.com

# pcs stonith sbd device message
/dev/mapper/36005076812808157b80000000000124b hdbnb1 clear# pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbna1: YES | YES | YES
hdbnb1: YES | YES | NO

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 clear
1 hdbnb1 clear hdbnb1.isicc.de.ibm.com

# pcs cluster start
Starting Cluster…

Check that the second node joined the cluster and try to fence the first node. On the second node, run the following commands.

# pcs status
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 15:17:15 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbna1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 15:17:16 2024
* Last change: Mon Aug 5 14:59:59 2024 by root via cibadmin on hdbna1
* 2 nodes configured
* 1 resource instance configured

Node List:
* Online: [ hdbna1 hdbnb1 ]

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbna1

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled

# pcs stonith fence hdbna1
Node: hdbna1 fenced# pcs status
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 15:23:21 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbnb1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 15:23:22 2024
* Last change: Mon Aug 5 14:59:59 2024 by root via cibadmin on hdbna1
* 2 nodes configured
* 1 resource instance configured

Node List:
* Online: [ hdbnb1 ]
* OFFLINE: [ hdbna1 ]

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbnb1

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled

The first node reboots. Wait until it is running, log in to the node, and then try to start the cluster without first clearing the SBD device.

# pcs cluster start
Starting Cluster…
Error: Unable to start pacemaker: A dependency job for pacemaker.service failed. See ‘journalctl -xe’ for details.

Because the SBD device is configured with the SBD_STARTMODE=clean option, you must first clear the SBD device before you can start the cluster. You can run the listed journalctl command to get more information about this error.

Clear the SBD device, start the cluster on the node, and check the cluster status.

# pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbnb1: YES | YES | YES
hdbna1: YES | YES | NO

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 reset hdbnb1.isicc.de.ibm.com
1 hdbnb1 clear

# pcs stonith sbd device message
/dev/mapper/36005076812808157b80000000000124b hdbna1 clear# pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbna1: YES | YES | NO
hdbnb1: YES | YES | YES

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 clear hdbna1.isicc.de.ibm.com
1 hdbnb1 clear# pcs cluster start
Starting Cluster…# pcs status
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 15:34:51 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbnb1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 15:34:51 2024
* Last change: Mon Aug 5 14:59:59 2024 by root via cibadmin on hdbna1
* 2 nodes configured
* 1 resource instance configured

Node List:
* Online: [ hdbna1 hdbnb1 ]

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbnb1

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled

Configuring the cluster to make an SAP solution highly available

At this point, the setup of the base cluster is complete.
From here, you can continue with the standard documentation to configure the cluster for one of the supported SAP high availability scenarios. The Red Hat HA Solutions for SAP HANA, S/4HANA and NetWeaver based SAP Applications provides references to documentation for implementing the supported scenarios.

A comprehensive set of documentation for implementing high availability for a range of SAP application scenarios is also available at Implementing High Availability for SAP Applications on IBM Power Virtual Server. Although this documentation is written for implementing a high-availability cluster running on IBM Power Virtual Server in the IBM Cloud, the cluster configuration of the SAP application scenarios is the same whether implemented in a cloud or in an on-premises environment. The main difference is the setup of the base cluster.

The following output shows the final cluster status of an SAP HANA System Replication cluster after following the instructions in Configuring SAP HANA Scale-Up System Replication in a RHEL HA Add-On cluster.

# pcs status –full
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-06 11:17:53 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbna1 (1) (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Tue Aug 6 11:17:54 2024
* Last change: Tue Aug 6 11:17:34 2024 by root via crm_attribute on hdbna1
* 2 nodes configured
* 6 resource instances configured

Node List:
* Node hdbna1 (1): online, feature set 3.16.2
* Node hdbnb1 (2): online, feature set 3.16.2

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbna1
* Clone Set: SAPHanaTopology_HDB_00-clone [SAPHanaTopology_HDB_00]:
* SAPHanaTopology_HDB_00 (ocf:heartbeat:SAPHanaTopology): Started hdbna1
* SAPHanaTopology_HDB_00 (ocf:heartbeat:SAPHanaTopology): Started hdbnb1
* Clone Set: SAPHana_HDB_00-clone [SAPHana_HDB_00] (promotable):
* SAPHana_HDB_00 (ocf:heartbeat:SAPHana): Promoted hdbna1
* SAPHana_HDB_00 (ocf:heartbeat:SAPHana): Unpromoted hdbnb1
* vip_HDB_00 (ocf:heartbeat:IPaddr2): Started hdbna1

Node Attributes:
* Node: hdbna1 (1):
* hana_hdb_clone_state : PROMOTED
* hana_hdb_op_mode : logreplay
* hana_hdb_remoteHost : hdbnb1
* hana_hdb_roles : 4:P:master1:master:worker:master
* hana_hdb_site : SiteA
* hana_hdb_srah : –
* hana_hdb_srmode : syncmem
* hana_hdb_sync_state : PRIM
* hana_hdb_version : 2.00.079.00
* hana_hdb_vhost : hdbna1
* lpa_hdb_lpt : 1722935854
* master-SAPHana_HDB_00 : 150
* Node: hdbnb1 (2):
* hana_hdb_clone_state : DEMOTED
* hana_hdb_op_mode : logreplay
* hana_hdb_remoteHost : hdbna1
* hana_hdb_roles : 4:S:master1:master:worker:master
* hana_hdb_site : SiteB
* hana_hdb_srah : –
* hana_hdb_srmode : syncmem
* hana_hdb_sync_state : SOK
* hana_hdb_version : 2.00.079.00
* hana_hdb_vhost : hdbnb1
* lpa_hdb_lpt : 30
* master-SAPHana_HDB_00 : 100

Migration Summary:

Tickets:

PCSD Status:
hdbna1: Online
hdbnb1: Online

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled

Summary

This blog post describes a sample implementation of a RHEL HA Add-On cluster on IBM Power10 servers using SBD fencing. SBD fencing simplifies the cluster setup and operation and is the recommended fencing method where available.
The resulting basic cluster implementation, combined with the proven reliability of IBM Power, provides the perfect platform for making a wide range of SAP application scenarios highly available.

IntroductionRed Hat Enterprise Linux (RHEL) for SAP provides a range of solutions for building highly available SAP environments for SAP applications. See Red Hat HA Solutions for SAP HANA, S/4HANA and NetWeaver based SAP Applications for an overview of available scenarios.This blog describes a sample setup of a RHEL HA Add-On cluster on IBM Power10 servers that can be used as a basis for implementing high availability scenarios for SAP applications. The cluster uses SBD (STONITH Block Device) for node fencing.The cluster uses IBM Power10 server partitions (LPARs) as cluster nodes. The information describes installing the high availability packages on each cluster node, setting up the cluster, and configuring the fencing devices.Target audience: Architects and specialists who are planning or implementing a high availability deployment of SAP applications on IBM Power10 servers.Conventions for shell examples later in this blog: All commands are executed as root and start with the # sign as the root prompt. Some commands use the continuation character to break the command into two or more lines. The output of some commands is omitted and replaced with three dots (…) instead.Cluster node fencing on IBM Power serversSTONITH is an acronym for Shoot The Other Node In The Head and protects your data from corruption in a split-brain situation. A robust fencing mechanism is essential for a cluster to prevent split-brain problems. You must enable STONITH (fencing) for a RHEL HA Add-On production cluster.SBD implements a fencing mechanism through exchanging messages via shared disk devices. SBD-based fencing requires a suitable watchdog timer device. The IBM PowerVM hypervisor implements such a watchdog timer service for Power10 processor-based servers with firmware level MH1030 or higher.For more information about SBD-based fencing, see the Red Hat Knowledge Base article Design Guidance for RHEL High Availability Clusters – sbd Considerations.The fence agents fence_sbd and fence_lpar are the two supported fencing agents for implementing STONITH devices on IBM Power servers. fence_lpar provides an alternative fencing mechanism that uses the HMC (Hardware Management Console) and is also supported on older IBM Power servers. For more information, see the Red Hat Knowledge Base article How can I configure Power fencing for the IBM Power platform using an HMC in a RHEL High Availability cluster?It is recommended to use fence_sbd if possible, as it simplifies the setup and also the operational behaviour.PrerequisitesThe prerequisites for using SBD-based fencing in LPARs running on IBM Power servers are:IBM Power10 servers with firmware level MH1030 or higher (see Power10 System Firmware Fix History)Cluster nodes must run in Power10 compatibility modeRed Hat Enterprise Linux version 9.2 or higherCluster nodes installed and configured according to the requirements of the SAP applicationOne or more shared storage devices exclusive to the STONITH configurationChecking the processor type and compatibility modeOn both nodes, run the following command to verify that the processor compatibility mode is set to Power10. The AT_BASE_PLATFORM field shows the actual processor type and the AT_PLATFORM field shows the processor compatibility mode. The processor compatibility mode must be set to Power10. Otherwise, the hypervisor watchdog won’t be available. # LD_SHOW_AUXV=1 /bin/true | grep _PLATFORM
AT_PLATFORM: power10
AT_BASE_PLATFORM: power10 Checking the Power10 hypervisor watchdogOn both nodes, run the following commands to verify that the Power10 hypervisor watchdog driver is loaded and the watchdog device is available. If one or both commands fail, make sure that the servers and LPARs meet the requirements documented above. # lsmod | grep pseries_wdt
pseries_wdt 262144 0  # wdctl /dev/watchdog
Device: /dev/watchdog
Identity: pseries-wdt [version 0]
Timeout: 60 seconds
Pre-timeout: 0 seconds
FLAG DESCRIPTION STATUS BOOT-STATUS
KEEPALIVEPING Keep alive ping reply 1 0
MAGICCLOSE Supports magic close char 0 0
PRETIMEOUT Pretimeout (in seconds) 0 0
SETTIMEOUT Set timeout (in seconds) 0 0 Installing and configuring a RHEL HA Add-On clusterUse the following steps to set up a two-node cluster on IBM Power10 servers. You must perform some steps on both nodes and some steps on either cluster node.Installing the RHEL HA Add-On softwareInstall the software packages that are required for the implementation of a cluster.Checking the RHEL HA repositoryOn both nodes, use the dnf repolist command to check the configured repositories. If the high-availability repository is missing, use the subscription-manager command to enable it. # dnf repolist
Updating Subscription Management repositories.
repo id repo name
rhel-9-for-ppc64le-appstream-e4s-rpms Red Hat Enterprise Linux 9 for Power, little endian – AppStream – Update Services for SAP Solutions (RPMs)
rhel-9-for-ppc64le-baseos-e4s-rpms Red Hat Enterprise Linux 9 for Power, little endian – BaseOS – Update Services for SAP Solutions (RPMs)
rhel-9-for-ppc64le-highavailability-e4s-rpms Red Hat Enterprise Linux 9 for Power, little endian – High Availability – Update Services for SAP Solutions (RPMs)
rhel-9-for-ppc64le-sap-solutions-e4s-rpms Red Hat Enterprise Linux 9 for Power, little endian – SAP Solutions – Update Services for SAP Solutions (RPMs)  # subscription-manager repos –enable=”rhel-9-for-ppc64le-highavailability-e4s-rpms”
Repository ‘rhel-9-for-ppc64le-highavailability-e4s-rpms’ is enabled for this system.# dnf clean all
Updating Subscription Management repositories.
46 files removed Installing the RHEL HA Add-On software packagesOn both nodes, run the following command to install the necessary software packages. # dnf -y install pcs pacemaker sbd fence-agents-sbd
… Configuring a RHEL HA Add-On clusterConfiguring the firewall servicesOn both nodes, run the following commands to add the high availability service to the RHEL firewall if firewalld.service is installed and enabled. # firewall-cmd –permanent –add-service=high-availability
success  # firewall-cmd –reload
success Starting the PCS daemonStart the PCS daemon that is used for controlling and configuring RHEL HA Add-On clusters through PCS. On both nodes, run the following commands to enable the service and then verify that it is running # systemctl enable –now pcsd.service
Created symlink /etc/systemd/system/multi-user.target.wants/pcsd.service → /usr/lib/systemd/system/pcsd.service.  # systemctl status pcsd.service
● pcsd.service – PCS GUI and remote configuration interface
Loaded: loaded (/usr/lib/systemd/system/pcsd.service; enabled; preset: disabled)
Active: active (running) since Mon 2024-08-05 13:05:32 CEST; 17s ago
Docs: man:pcsd(8)
man:pcs(8)
Main PID: 42235 (pcsd)
Tasks: 29 (limit: 312708)
Memory: 248.3M
CPU: 1.963s
CGroup: /system.slice/pcsd.service
├─42235 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42238 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42248 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42249 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42251 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42255 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42258 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42263 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42267 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42271 /usr/bin/python3 -Es /usr/sbin/pcsd
├─42277 /usr/bin/python3 -Es /usr/sbin/pcsd
└─42282 /usr/bin/python3 -Es /usr/sbin/pcsd

Aug 05 13:05:30 hdbna1.isicc.de.ibm.com systemd[1]: Starting PCS GUI and remote configuration interface…
Aug 05 13:05:32 hdbna1.isicc.de.ibm.com systemd[1]: Started PCS GUI and remote configuration interface. Populating entries for each node in the hosts fileOn both nodes, add the IP addresses and host names of both hosts to the /etc/hosts file. For more information, see Setting up /etc/hosts files on RHEL cluster nodes.Setting a password for the hacluster user IDOn both nodes, run the passwd command and enter a password for the hacluster user ID. # passwd hacluster
Changing password for user hacluster.
New password:
Retype new password:
passwd: all authentication tokens updated successfully. Authenticating the cluster nodesUse the following command to authenticate user hacluster to the PCS daemon on the nodes in the cluster. The command prompts you for the password you set in the previous step. On any node, run the following command. # pcs host auth hdbna1 hdbnb1 -u hacluster
Password:
hdbnb1: Authorized
hdbna1: Authorized Configuring and starting the cluster nodesConfigure the cluster configuration file and synchronise the configuration to the specified nodes. The –start option also starts the cluster service on the nodes. On any node, run the following commands to set up the cluster and then check the status of the new cluster. # pcs cluster setup SAP_CLUSTER –start hdbna1 hdbnb1
No addresses specified for host ‘hdbna1’, using ‘hdbna1’
No addresses specified for host ‘hdbnb1’, using ‘hdbnb1’
Destroying cluster on hosts: ‘hdbna1’, ‘hdbnb1’…
hdbna1: Successfully destroyed cluster
hdbnb1: Successfully destroyed cluster
Requesting remove ‘pcsd settings’ from ‘hdbna1’, ‘hdbnb1’
hdbna1: successful removal of the file ‘pcsd settings’
hdbnb1: successful removal of the file ‘pcsd settings’
Sending ‘corosync authkey’, ‘pacemaker authkey’ to ‘hdbna1’, ‘hdbnb1’
hdbna1: successful distribution of the file ‘corosync authkey’
hdbna1: successful distribution of the file ‘pacemaker authkey’
hdbnb1: successful distribution of the file ‘corosync authkey’
hdbnb1: successful distribution of the file ‘pacemaker authkey’
Sending ‘corosync.conf’ to ‘hdbna1’, ‘hdbnb1’
hdbna1: successful distribution of the file ‘corosync.conf’
hdbnb1: successful distribution of the file ‘corosync.conf’
Cluster has been successfully set up.
Starting cluster on hosts: ‘hdbna1’, ‘hdbnb1’… # pcs cluster status
Cluster Status:
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 13:31:51 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbna1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 13:31:51 2024
* Last change: Mon Aug 5 13:30:49 2024 by hacluster via crmd on hdbna1
* 2 nodes configured
* 0 resource instances configured
Node List:
* Online: [ hdbna1 hdbnb1 ]

PCSD Status:
hdbna1: Online
hdbnb1: Online Configuring the cluster fencing deviceUse the following information to create the mandatory fencing device.Validating the watchdog timer device to use with sbdYou can use the following command to verify that the hypervisor watchdog works properly. The command causes the node to reset after a short countdown. # sbd -w /dev/watchdog test-watchdog

WARNING: This operation is expected to force-reboot this system
without following any shutdown procedures.

Proceed? [NO/Proceed] Proceed

Initializing /dev/watchdog with a reset countdown of 5 seconds …

NOTICE: The watchdog device is expected to reset the system
in 5 seconds. If system remains active beyond that time,
watchdog may not be functional.

Reset countdown … 5 seconds
Reset countdown … 4 seconds
Reset countdown … 3 seconds
Reset countdown … 2 seconds
System expected to reset any moment … For more information, see the Red Hat support article Administrative Procedures for RHEL High Availability Clusters – Validating a Watchdog Timer Device (WDT) to Use with sbd.Getting the WWID of the STONITH SBD devicesTo configure SBD STONITH, you need one or more shared storage devices. Use pvs –all or multipath -l and note the WWIDs of the shared disk devices that are assigned to both LPARs.The following example shows a setup with a single SBD device.Creating an SBD deviceOn any node, run the following commands to set up and then enable the SBD device.  # pcs stonith sbd device setup
device=/dev/mapper/36005076812808157b80000000000124b
WARNING: All current content on device(s) ‘/dev/mapper/36005076812808157b80000000000124b’ will be overwritten
Type ‘yes’ or ‘y’ to proceed, anything else to cancel: y
Initializing device ‘/dev/mapper/36005076812808157b80000000000124b’…
Device initialized successfully  # pcs stonith sbd enable
device=/dev/mapper/36005076812808157b80000000000124b
SBD_WATCHDOG_TIMEOUT=10 SBD_STARTMODE=clean
Running SBD pre-enabling checks…
hdbnb1: SBD pre-enabling checks done
hdbna1: SBD pre-enabling checks done
Distributing SBD config…
hdbna1: SBD config saved
hdbnb1: SBD config saved
Enabling sbd…
hdbna1: sbd enabled
hdbnb1: sbd enabled
Warning: Cluster restart is required in order to apply these changes. Checking the SBD configurationCheck the contents of the SBD configuration file. Since the configuration file was generated by pcs, the contents should be the same on both nodes.  # cat /etc/sysconfig/sbd
# This file has been generated by pcs.
SBD_DELAY_START=no
SBD_DEVICE=”/dev/mapper/36005076812808157b80000000000124b”
SBD_OPTS=”-n hdbna1″
SBD_PACEMAKER=yes
SBD_STARTMODE=clean
SBD_WATCHDOG_DEV=/dev/watchdog
SBD_WATCHDOG_TIMEOUT=10 The following commands show some more details about the available watchdog devices. # pcs stonith sbd watchdog list
Available watchdog(s):
/dev/watchdog
/dev/watchdog0  # sbd query-watchdog

Discovered 2 watchdog devices:

[1] /dev/watchdog
Identity: pseries-wdt
Driver: pseries-wdt

[2] /dev/watchdog0
Identity: pseries-wdt
Driver: pseries-wdt# wdctl /dev/watchdog
Device: /dev/watchdog
Identity: pseries-wdt [version 0]
Timeout: 60 seconds
Pre-timeout: 0 seconds
FLAG DESCRIPTION STATUS BOOT-STATUS
KEEPALIVEPING Keep alive ping reply 1 0
MAGICCLOSE Supports magic close char 1 0
PRETIMEOUT Pretimeout (in seconds) 0 0
SETTIMEOUT Set timeout (in seconds) 0 0# wdctl /dev/watchdog0
Device: /dev/watchdog0
Identity: pseries-wdt [version 0]
Timeout: 60 seconds
Pre-timeout: 0 seconds
FLAG DESCRIPTION STATUS BOOT-STATUS
KEEPALIVEPING Keep alive ping reply 1 0
MAGICCLOSE Supports magic close char 0 0
PRETIMEOUT Pretimeout (in seconds) 0 0
SETTIMEOUT Set timeout (in seconds) 0 0 Testing the watchdog with a stonith operationOn both nodes, run the following command to verify that the stonith operation forces the node to reboot. # pcs stonith sbd watchdog test /dev/watchdog
WARNING: This operation is expected to force-reboot this system without following any shutdown procedures
Type ‘yes’ or ‘y’ to proceed, anything else to cancel: yes
System will reset shortly Creating the STONITH deviceThe following command shows the device-specific options for the fence_sbd fencing agent. Alternatively, you can check its man pages with man fence_sbd. # pcs stonith describe fence_sbd
… On any node, start the cluster and then run the pcs stonith create command to create the STONITH device. # pcs cluster start –all
hdbna1: Starting Cluster…
hdbnb1: Starting Cluster… # pcs stonith create res_fence_sbd fence_sbd
devices=/dev/mapper/36005076812808157b80000000000124b Verifying the cluster configurationUse the following commands to display the STONITH configuration and the status of the SBD device. # pcs stonith config
Resource: res_fence_sbd (class=stonith type=fence_sbd)
Attributes: res_fence_sbd-instance_attributes
devices=/dev/mapper/36005076812808157b80000000000124b
Operations:
monitor: res_fence_sbd-monitor-interval-60s
interval=60s # pcs stonith sbd config
SBD_DELAY_START=no
SBD_STARTMODE=clean
SBD_WATCHDOG_TIMEOUT=10

Watchdogs:
hdbna1: /dev/watchdog
hdbnb1: /dev/watchdog

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 clear
1 hdbnb1 clear

SBD header on device ‘/dev/mapper/36005076812808157b80000000000124b’:
==Dumping header on disk /dev/mapper/36005076812808157b80000000000124b
Header version : 2.1
UUID : c0602a56-64c2-4a42-9d21-8ac0e67b2b46
Number of slots : 255
Sector size : 512
Timeout (watchdog) : 5
Timeout (allocate) : 2
Timeout (loop) : 1
Timeout (msgwait) : 10
==Header on disk /dev/mapper/36005076812808157b80000000000124b is dumped Testing the fencing of the nodesTo test the STONITH configuration, you must manually fence the nodes. On the first node, run the following commands to fence the second node and then view the cluster status. # pcs stonith fence hdbnb1
Node: hdbnb1 fenced # pcs status
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 15:07:41 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbna1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 15:07:41 2024
* Last change: Mon Aug 5 14:59:59 2024 by root via cibadmin on hdbna1
* 2 nodes configured
* 1 resource instance configured

Node List:
* Online: [ hdbna1 ]
* OFFLINE: [ hdbnb1 ]

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbna1

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled The command has resulted in a reboot of the second node.After the reboot, log in to the second node. Then, you must clear the SBD device before starting the cluster. # pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbna1: YES | YES | YES
hdbnb1: YES | YES | NO

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 clear
1 hdbnb1 reset hdbna1.isicc.de.ibm.com # pcs stonith sbd device message
/dev/mapper/36005076812808157b80000000000124b hdbnb1 clear# pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbna1: YES | YES | YES
hdbnb1: YES | YES | NO

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 clear
1 hdbnb1 clear hdbnb1.isicc.de.ibm.com

# pcs cluster start
Starting Cluster… Check that the second node joined the cluster and try to fence the first node. On the second node, run the following commands. # pcs status
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 15:17:15 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbna1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 15:17:16 2024
* Last change: Mon Aug 5 14:59:59 2024 by root via cibadmin on hdbna1
* 2 nodes configured
* 1 resource instance configured

Node List:
* Online: [ hdbna1 hdbnb1 ]

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbna1

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled # pcs stonith fence hdbna1
Node: hdbna1 fenced# pcs status
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-05 15:23:21 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbnb1 (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Mon Aug 5 15:23:22 2024
* Last change: Mon Aug 5 14:59:59 2024 by root via cibadmin on hdbna1
* 2 nodes configured
* 1 resource instance configured

Node List:
* Online: [ hdbnb1 ]
* OFFLINE: [ hdbna1 ]

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbnb1

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled The first node reboots. Wait until it is running, log in to the node, and then try to start the cluster without first clearing the SBD device. # pcs cluster start
Starting Cluster…
Error: Unable to start pacemaker: A dependency job for pacemaker.service failed. See ‘journalctl -xe’ for details. Because the SBD device is configured with the SBD_STARTMODE=clean option, you must first clear the SBD device before you can start the cluster. You can run the listed journalctl command to get more information about this error.Clear the SBD device, start the cluster on the node, and check the cluster status. # pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbnb1: YES | YES | YES
hdbna1: YES | YES | NO

Messages list on device ‘/dev/mapper/36005076812808157b80000000000124b’:
0 hdbna1 reset hdbnb1.isicc.de.ibm.com
1 hdbnb1 clear # pcs stonith sbd device message
/dev/mapper/36005076812808157b80000000000124b hdbna1 clear# pcs stonith sbd status
SBD STATUS
<node name>: <installed> | <enabled> | <running>
hdbna1: YES | YES | NO
hdbnb1: YES | YES | YES

Node List:
* Online: [ hdbna1 hdbnb1 ]

Full List of Resources:
* res_fence_sbd (stonith:fence_sbd): Started hdbnb1

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled Configuring the cluster to make an SAP solution highly availableAt this point, the setup of the base cluster is complete.From here, you can continue with the standard documentation to configure the cluster for one of the supported SAP high availability scenarios. The Red Hat HA Solutions for SAP HANA, S/4HANA and NetWeaver based SAP Applications provides references to documentation for implementing the supported scenarios.A comprehensive set of documentation for implementing high availability for a range of SAP application scenarios is also available at Implementing High Availability for SAP Applications on IBM Power Virtual Server. Although this documentation is written for implementing a high-availability cluster running on IBM Power Virtual Server in the IBM Cloud, the cluster configuration of the SAP application scenarios is the same whether implemented in a cloud or in an on-premises environment. The main difference is the setup of the base cluster.The following output shows the final cluster status of an SAP HANA System Replication cluster after following the instructions in Configuring SAP HANA Scale-Up System Replication in a RHEL HA Add-On cluster. # pcs status –full
Cluster name: SAP_CLUSTER
Status of pacemakerd: ‘Pacemaker is running’ (last updated 2024-08-06 11:17:53 +02:00)
Cluster Summary:
* Stack: corosync
* Current DC: hdbna1 (1) (version 2.1.5-9.el9_2.4-a3f44794f94) – partition with quorum
* Last updated: Tue Aug 6 11:17:54 2024
* Last change: Tue Aug 6 11:17:34 2024 by root via crm_attribute on hdbna1
* 2 nodes configured
* 6 resource instances configured

Node List:
* Node hdbna1 (1): online, feature set 3.16.2
* Node hdbnb1 (2): online, feature set 3.16.2

Migration Summary:

Tickets:

PCSD Status:
hdbna1: Online
hdbnb1: Online

Daemon Status:
corosync: active/disabled
pacemaker: active/disabled
pcsd: active/enabled
sbd: active/enabled SummaryThis blog post describes a sample implementation of a RHEL HA Add-On cluster on IBM Power10 servers using SBD fencing. SBD fencing simplifies the cluster setup and operation and is the recommended fencing method where available.The resulting basic cluster implementation, combined with the proven reliability of IBM Power, provides the perfect platform for making a wide range of SAP application scenarios highly available. Read More Technology Blogs by Members articles

#SAP

#SAPTechnologyblog