Recover a Hot Standby Redundancy System After a Site Loss

About This Procedure

This procedure describes how to recover a Hot Standby Redundancy system after one of its two sites is permanently lost.

Key Terms

The following key terms are used throughout this article:

• Surviving site: The Hot Standby Redundancy system site that stays online and contains the latest database.
• Source site: The original site that was lost and cannot be recovered.
• Replacement site: The new site that replaces the source site.

Note: The replacement site must take the same site role as the source site. For example, if site-b was lost, the replacement becomes site-b.

Procedure Prerequisites

1. The surviving site must be active and contain the latest database before starting this procedure.
2. You have the skylight user credentials.
3. You have a scheduled maintenance window.
4. The source site is confirmed to be unrecoverable (so this procedure is not run during a transient outage). Keep the lost site powered off or isolated from the network. If it comes back with the same IP addresses or MAC addresses, it can cause split-brain or overwrite the latest database.

Procedure Steps

Back Up the Latest Database

Back up the database before you proceed.

To back up the latest database:

1. Open an SSH terminal session to the Legacy Orchestrator CLI on the surviving site using port 2200.

2. Log in as the skylight user.

3. Create a backup of the data store by entering:

   database backup
   

   Note: The backup file is saved as a .tar.gz file in /home/skylight/so/mysql-ha/backups.

4. Transfer the backup file to another server to preserve the latest database backup file.

Stop Hot Standby Redundancy on the Surviving Site

1. Open an SSH terminal session to the Legacy Orchestrator CLI on the surviving site using port 2200.

2. Log in as the skylight user.

3. Stop Hot Standby Redundancy by entering:

   redundancy control stop
   

   Note: An HA_DEACTIVATION_FAILED error is expected because the source site can no longer be contacted.

4. Configure the surviving site as the preferred site to ensure that there will be no loss to the latest database when Hot Standby Redundancy starts again by entering:

   redundancy config preferred {site-a | site-b}
   

Verify the Surviving Site After Stopping Hot Standby Redundancy

Verify the following on the surviving site after stopping Hot Standby Redundancy to ensure that no further issues arise.

At the Legacy Orchestrator CLI

1. Open an SSH terminal session to the Legacy Orchestrator CLI on the surviving site using port 2200.

2. Log in as the skylight user.

3. Check Hot Standby Redundancy status by entering:

   redundancy show status
   

   Expectation: "Global Status" should be "Stopped".

4. Check Hot Standby Redundancy configuration by entering:

   redundancy show configuration
   

   Expectation: "Preferred site" should be the surviving site (whether it's site-a or site-b depends on the actual circumstances of the system).

At the Docker host

1. Open an SSH terminal session to the surviving site using port 22.

2. Log in as the skylight user.

3. Check drbd0 partition status by entering:

   lsblk
   

   Expectation: there is no information regarding a 'drbd0' partition or any mount point associated with 'mysql-ha'.

Prepare the Replacement Site

If an OS backup of the source site exists, restoring it saves configuration steps. If not, deploy from scratch. There are two options:

Option 1: Deploy the Replacement Site if the Backup from the Source Site Is Unavailable

1. Deploy a new Docker host for the replacement site.
2. Install the Legacy Orchestrator Docker at the same version as the surviving site. For details, see Legacy Orchestrator Docker Deployment.
3. Configure and update the Hot Standby Redundancy configuration. For details, see Introduction to Hot Standby Redundancy and the Hot Standby-related pages in the Administrative Operations subcategory.

IMPORTANT: Do not start Hot Standby Redundancy yet. Complete the following steps before starting.

Option 2: Restore the Backup from the Source Site if Available

1. Restore the backup to the replacement host.

2. At the first boot, open an SSH terminal session to the Legacy Orchestrator CLI on the replacement site using port 2200.

3. Log in as the skylight user.

4. Check Hot Standby Redundancy status by entering:

   redundancy show status
   

   The replacement site is running with Active, Passive or Offline state; and the surviving site is stopped (Offline state)

5. If Hot Standby Redundancy is still running (Active or Passive state), stop it by entering:

   redundancy control stop
   

Verify the Replacement Site After Stopping Hot Standby Redundancy (Option 2 Only)

Verify the following on the replacement site after stopping Hot Standby Redundancy to ensure that no further issues arise.

At the Legacy Orchestrator CLI

1. Open an SSH terminal session to the Legacy Orchestrator CLI on the replacement site using port 2200.

2. Log in as the skylight user.

3. Check Hot Standby Redundancy status by entering:

   redundancy show status
   

   Expectation: "Global Status" should be "Stopped".

At the Docker host

1. Open an SSH terminal session to the replacement site using port 22.

2. Log in as the skylight user.

3. Check drbd0 partition status by entering:

   lsblk
   

   Expectation: there is no information regarding a 'drbd0' partition or any mount point associated with 'mysql-ha'.

Verify and Update the Replacement Site

1. Verify network card MAC addresses.

   The network card MAC addresses on the replacement site must match those on the source site or the addresses specified for the source site in the Hot Standby Redundancy license file.
   If they don't match, reconfigure the MAC address on the replacement site or relicense with the new MAC address.

2. Verify the replication partition size.
   The total size of a block device (the replication partition) in bytes must match on the replacement site and the surviving site.

   sudo blockdev --getsize64 <the replication partition>
   

   For example:
   sudo blockdev --getsize64 /dev/sdb
   If the sizes differ, resize the replication partition on the replacement site before continuing.

3. Verify the timezone and NTP synchronization configuration.
   Refer to the commands below to verify:

   timedatectl
   timedatectl show -p Timezone -p NTPSynchronized
   cat /etc/timezone
   

   They must match on the replacement and surviving sites. If they differ, reconfigure the timezone and NTP synchronization before continuing.
   For timezone changes, see the "Changing Timezone Configuration" section in Legacy Orchestrator Docker Deployment. For NTP synchronization, follow the standard configuration guidelines for the Docker host OS.

4. Update and verify the Hot Standby Redundancy configuration.
   For details, see Configure Hot Standby Redundancy.

   1. Reconfigure the replication partition on the replacement site
      This step is required when an OS backup was restored and the replication partition configuration is no longer valid. Verify that the selected partition contains no required data and is not configured as a mounted file system.
   2. Reimport the Hot Standby Redundancy license on the replacement site.
   3. The hostname, IP addresses for replication and monitoring links, and connections between the replacement site and the surviving site.
   4. The replacement site information in the Hot Standby Redundancy configuration on the surviving site must match the current replacement site.

Start Hot Standby Redundancy on the Surviving Site

Before starting Hot Standby Redundancy: Docker 29.2.0+ check

If Docker version 29 (29.2.0+) or later is in use on the system, and the Legacy Orchestrator services were started in a standalone (non-HA) environment for a quick check, remove all containers of Legacy Orchestrator services before starting the Hot Standby Redundancy.

Follow only the container-removal steps in the "Use Case #2: Transitioning to HA-Managed Environment" section in Docker 29.2.0 and Hot Standby Redundancy Feature Incompatibility. Skip the start step in that article.

Start Hot Standby Redundancy on the surviving site

1. Open an SSH terminal session to the Legacy Orchestrator CLI on the surviving site using port 2200.

2. Log in as the skylight user.

3. Start Hot Standby Redundancy by entering:

   redundancy control start
   

Verify Hot Standby Redundancy Status After Starting

Verify the following on both sites after starting Hot Standby Redundancy to ensure that no further issues arise.

At the Legacy Orchestrator CLI on the surviving site

1. Open an SSH terminal session to the Legacy Orchestrator CLI on the surviving site using port 2200.

2. Log in as the skylight user.

3. Check Hot Standby Redundancy status by entering:

   redundancy show status
   

   Expectation:

• The "Global Status" should be "Started"
• The surviving site should be "Active" state and the replacement site should be "Passive" state
• The "Replication status" status should be "Synchronizing", and will change to "Up" once the synchronization is complete.

4. Check Hot Standby Redundancy configuration by entering:

   redundancy show configuration
   

   Expectation: "Preferred site" should be the surviving site (whether it's site-a or site-b depends on the actual circumstances of the system).

At the Legacy Orchestrator CLI on the replacement site

The steps and expectations are similar to "Verify at the Legacy Orchestrator CLI on the surviving site" section above.

At the Docker host on the surviving site

1. Open an SSH terminal session to the surviving site using port 22.

2. Log in as the skylight user.

3. Check drbd0 partition status by entering:

   lsblk
   

   Expectation: the 'drbd0' partition will be configured below the replicated partition and a mount point associated with 'mysql-ha'

   Example:

   $lsblk
   ...
   sdb       8:16   0   30G  0 disk 
   └─drbd0 147:0    0   30G  0 disk /home/skylight/24.09.1-GA/mysql-ha
   

At the Docker host on the replacement site

1. Open an SSH terminal session to the replacement site using port 22.

2. Log in as the skylight user.

3. Check drbd0 partition status by entering:

   lsblk
   

   Expectation: the 'drbd0' partition will be configured below the replicated partition and there is no mount point associated with 'mysql-ha'

   Example:

   $lsblk
   ...
   sdb       8:16   0   30G  0 disk 
   └─drbd0 147:0    0   30G  0 disk
   

Update the Preferred Site to Its Original Configuration After Starting Hot Standby Redundancy

This step is optional. After starting Hot Standby Redundancy, follow this step to update the preferred site to its original state.

1. Open an SSH terminal session to the Legacy Orchestrator CLI on the surviving site using port 2200.

2. Log in as the skylight user.

3. Check the Hot Standby Redundancy status by entering:

   redundancy show status
   

   Expectation: The "Replication status" must be "Up". If it still shows "Synchronizing", wait until it changes to "Up".

4. Update the preferred site to its original configuration by entering:

   redundancy control set-preferred-site {site-a | site-b}