Configuring H/W Sensor Management - Hot Standby System

Introduction

This article provides the configuration values and verification steps for deploying Hardware Sensor Management (H/W SM) in a Hot Standby Redundancy environment with two worker VM instances. Use this article alongside the Replicated Cluster with H/W Sensor Management Worker Deployment article, which contains the broader Crosswork Assurance deployment workflow and additional background details.

In a hot standby setup, the data store is replicated to the secondary system continuously at block level. Failover can be automatic.

For manual failover with periodic backups instead, see Configuring H/W Sensor Management - Warm Standby System

Initial Setup

The initial setup for Hot Standby Redundancy is as follows:

• Two H/W SM hosts run as Site A (Host A) and Site B (Host B). One site is Active and manages devices; the other is Passive, ready to replace the Active site in case of failure.
• The H/W Sensor Management data is replicated between the sites at block level (DRBD) over a dedicated replication network.
• A monitoring link between the sites detects failures. With Automatic Failover enabled, the Passive site takes over without manual action.
• You can also switch the active site manually, and suspend redundancy for maintenance.

Prerequisites Specific to Hot Standby

Both hosts must also meet the H/W Sensor Management Worker Node Requirements and the skylight user requirements on the Replicated Cluster with H/W Sensor Management Worker Deployment article.
In addition:

Details for Configuring H/W Sensor Management on the Crosswork Assurance Admin Console Web GUI

Complete all groups before you save the configuration for the first time.

Host Information

High Availability

Application Configuration

Leave ⚠ Reset Worker Node(s) (Destructive) off.

H/W Sensor Collector

Refer to Configuring the Sensor Collector for H/W and S/W Sensor Management for details.

Deploy and Verify H/W SM Workers

The H/W SM workers are deployed together with Crosswork Assurance in a single deployment. Pre-flight checks include the H/W SM worker checks (SSH as skylight, sudo, firewalld/SELinux, disk, and other prerequisites) and the HA checks below; resolve any failures before deploying.

1. Preflight Checks

Some HA checks are expected warnings before the first deployment, because the services they test are not installed yet:

2. Follow the Deployment

Follow the progress on the Controller node:

1. Access the root user or run sudo su - if the account has sudo privileges.
2. Configure to use kubectl command, enter:

export KUBECONFIG="/var/lib/embedded-cluster/k0s/pki/admin.conf"
export PATH="$PATH:/var/lib/embedded-cluster/bin"

3. Check pods and job log, enter:

kubectl -n pca get jobs,pods | grep hw-sm
kubectl -n pca logs job/<hw-sm-app-deploy-job-name>

When the installation succeeds, the hw-sm-app-deploy log ends with:

The installation of Cisco Crosswork Assurance Sensor Management <version> ... is complete

The H/W SM jobs run in a chain. The HA jobs run last, in this order:

1. ha-hot-license-import
2. ha-hot-block-device-prep
3. ha-hot-reconfigure

The full deployment can take about one hour or more (typical time by sizing). One Error pod per hw-sm-app-deploy job is a known harmless retry.

3. Verify the Deployment

Important: the ha-hot-reconfigure job can print HA_ACTIVATION_FAILED and still finish as Complete. Always read its log.

The H/W SM files are under /home/skylight/so/ on each worker. The HA log is /home/skylight/so/logs/hamon/hamon.log (root-owned).

Operations

• Switch the active site manually: increase Manual Switch Trigger, deploy.
• Suspend redundancy for maintenance: check Suspend Redundancy Acknowledgment, increase Suspend Redundancy (Maintenance Mode), deploy. To resume, increase Resume Redundancy and deploy.
• Reset the workers: see Resetting the H/W SM Worker Node. The reset runs on both hosts, removes all Sensor Management data, and has no backup. Enter a Reset Trigger number higher than any used before.

Known Limitations

• Console shows "Unavailable": the Admin Console can show the application as Unavailable while HA is healthy, because the hw-sm-status-host-b deployment is never Ready while Site B is Passive.