---
title: "Recover a Hot Standby Redundancy System After a Site Loss"
slug: "recover-a-hot-standby-redundancy-system-after-a-site-loss"
tags: ["Legacy Orchestrator", "Hot Standby Redundancy", "Docker"]
updated: 2026-11-03T14:42:49Z
published: 2026-11-03T14:42:49Z
canonical: "docs.crossworkassurance.cisco.com/recover-a-hot-standby-redundancy-system-after-a-site-loss"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://docs.crossworkassurance.cisco.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Recover a Hot Standby Redundancy System After a Site Loss

## About This Procedure

This procedure describes how to recover a Hot Standby Redundancy system after one of its two sites is permanently lost.

### Key Terms

The following key terms are used throughout this article:

- **Surviving site**: The Hot Standby Redundancy system site that stays online and contains the latest database.
- **Source site**: The original site that was lost and cannot be recovered.
- **Replacement site**: The new site that replaces the source site.

> **Note:** The replacement site must take the same site role as the source site. For example, if site-b was lost, the replacement becomes site-b.

## Procedure Prerequisites

1. The surviving site must be active and contain the latest database before starting this procedure.
2. You have the **skylight** user credentials.
3. You have a scheduled maintenance window.
4. The source site is confirmed to be unrecoverable (so this procedure is not run during a transient outage). Keep the lost site powered off or isolated from the network. If it comes back with the same IP addresses or MAC addresses, it can cause split-brain or overwrite the latest database.

Warning:

- Contact the technical support team if any prerequisite is not met
- Do not attempt to perform troubleshooting steps on the system yourself, as doing so could lead to further issues that might affect the system and the latest database

## Procedure Steps

### Back Up the Latest Database

Back up the database before you proceed.

**To back up the latest database:**

1. Open an SSH terminal session to the Legacy Orchestrator CLI on **the surviving site** using port 2200.
2. Log in as the **skylight** user.
3. Create a backup of the data store by entering:

```
database backup
```

> **Note:** The backup file is saved as a `.tar.gz` file in `/home/skylight/so/mysql-ha/backups`.
4. Transfer the backup file to another server to preserve the latest database backup file.

### Stop Hot Standby Redundancy on the Surviving Site

1. Open an SSH terminal session to the Legacy Orchestrator CLI on **the surviving site** using port 2200.
2. Log in as the **skylight** user.
3. Stop Hot Standby Redundancy by entering:

```
redundancy control stop
```

> **Note:** An `HA_DEACTIVATION_FAILED` error is expected because the source site can no longer be contacted.
4. Configure the surviving site as the preferred site to ensure that there will be no loss to the latest database when Hot Standby Redundancy starts again by entering:

```
redundancy config preferred {site-a | site-b}
```

### Verify the Surviving Site After Stopping Hot Standby Redundancy

Verify the following on the surviving site after stopping Hot Standby Redundancy to ensure that no further issues arise.

Warning:

- Contact the technical support team if there are any issues other than those described in this section
- Do not attempt to perform troubleshooting steps on the system yourself, as doing so could lead to further issues that might affect the system and the latest database

**At the Legacy Orchestrator CLI**

1. Open an SSH terminal session to the Legacy Orchestrator CLI on **the surviving site** using port 2200.
2. Log in as the **skylight** user.
3. Check Hot Standby Redundancy status by entering:

```
redundancy show status
```

**Expectation**: "Global Status" should be "Stopped".
4. Check Hot Standby Redundancy configuration by entering:

```
redundancy show configuration
```

Expectation: "Preferred site" should be the surviving site (whether it's site-a or site-b depends on the actual circumstances of the system).

**At the Docker host**

1. Open an SSH terminal session to **the surviving site** using port 22.
2. Log in as the **skylight** user.
3. Check drbd0 partition status by entering:

```
lsblk
```

Expectation: there is no information regarding a 'drbd0' partition or any mount point associated with 'mysql-ha'.

### Prepare the Replacement Site

If an OS backup of the source site exists, restoring it saves configuration steps. If not, deploy from scratch. There are two options:

#### Option 1: Deploy the Replacement Site if the Backup from the Source Site Is Unavailable

1. Deploy a new Docker host for the replacement site.
2. Install the Legacy Orchestrator Docker at the same version as the surviving site. For details, see [Legacy Orchestrator Docker Deployment](/docs/legacy-orchestrator-docker-deployment).
3. Configure and update the Hot Standby Redundancy configuration. For details, see [Introduction to Hot Standby Redundancy](/docs/hot-standby-redundancy)  and the Hot Standby-related pages in the *Administrative Operations* subcategory.

**IMPORTANT**: Do not start Hot Standby Redundancy yet. Complete the following steps before starting.

#### Option 2: Restore the Backup from the Source Site if Available

1. Restore the backup to the replacement host.
2. At the first boot, open an SSH terminal session to the Legacy Orchestrator CLI on **the replacement site** using port 2200.
3. Log in as the **skylight** user.
4. Check Hot Standby Redundancy status by entering:

```
redundancy show status
```

The replacement site is running with Active, Passive or Offline state; and the surviving site is stopped (Offline state)
5. If Hot Standby Redundancy is still running (Active or Passive state), stop it by entering:

```
redundancy control stop
```

#### Verify the Replacement Site After Stopping Hot Standby Redundancy (Option 2 Only)

Verify the following on the replacement site after stopping Hot Standby Redundancy to ensure that no further issues arise.

Warning:

- Contact the technical support team if there are any issues other than those described in this section
- Do not attempt to perform troubleshooting steps on the system yourself, as doing so could lead to further issues that might affect the system and the latest database

**At the Legacy Orchestrator CLI**

1. Open an SSH terminal session to the Legacy Orchestrator CLI on **the replacement site** using port 2200.
2. Log in as the **skylight** user.
3. Check Hot Standby Redundancy status by entering:

```
redundancy show status
```

**Expectation**: "Global Status" should be "Stopped".

**At the Docker host**

1. Open an SSH terminal session to **the replacement site** using port 22.
2. Log in as the **skylight** user.
3. Check drbd0 partition status by entering:

```
lsblk
```

**Expectation**: there is no information regarding a 'drbd0' partition or any mount point associated with 'mysql-ha'.

### Verify and Update the Replacement Site

1. Verify network card MAC addresses.

The network card MAC addresses on the replacement site must match those on the source site or the addresses specified for the source site in the Hot Standby Redundancy license file. If they don't match, reconfigure the MAC address on the replacement site or relicense with the new MAC address.
2. Verify the replication partition size. The total size of a block device (the replication partition) in bytes must match on the replacement site and the surviving site.

```
sudo blockdev --getsize64 <the replication partition>
```

**For example**: `sudo blockdev --getsize64 /dev/sdb` If the sizes differ, resize the replication partition on the replacement site before continuing.
3. Verify the timezone and NTP synchronization configuration. Refer to the commands below to verify:

```
timedatectl
timedatectl show -p Timezone -p NTPSynchronized
cat /etc/timezone
```

They must match on the replacement and surviving sites. If they differ, reconfigure the timezone and NTP synchronization before continuing. For timezone changes, see the "Changing Timezone Configuration" section in [Legacy Orchestrator Docker Deployment](/docs/legacy-orchestrator-docker-deployment). For NTP synchronization, follow the standard configuration guidelines for the Docker host OS.
4. Update and verify the Hot Standby Redundancy configuration. For details, see [Configure Hot Standby Redundancy](/docs/configuring-hot-standby-redundancy).
  1. Reconfigure the replication partition on the replacement site  

This step is required when an OS backup was restored and the replication partition configuration is no longer valid. Verify that the selected partition contains no required data and is not configured as a mounted file system.
  2. Reimport the Hot Standby Redundancy license on the replacement site.
  3. The hostname, IP addresses for replication and monitoring links, and connections between the replacement site and the surviving site.
  4. The replacement site information in the Hot Standby Redundancy configuration on the surviving site must match the current replacement site.

### Start Hot Standby Redundancy on the Surviving Site

**Before starting Hot Standby Redundancy: Docker 29.2.0+ check**

If Docker version 29 (29.2.0+) or later is in use on the system, and the Legacy Orchestrator services were started in a standalone (non-HA) environment for a quick check, remove all containers of Legacy Orchestrator services before starting the Hot Standby Redundancy.

Follow only the container-removal steps in the "Use Case #2: Transitioning to HA-Managed Environment" section in [Docker 29.2.0 and Hot Standby Redundancy Feature Incompatibility](/docs/docker-29-2-0-host-and-hot-standby-redundancy-feature-incompatibility-use-cases). Skip the start step in that article.

**Start Hot Standby Redundancy on the surviving site**

1. Open an SSH terminal session to the Legacy Orchestrator CLI on **the surviving site** using port 2200.
2. Log in as the **skylight** user.
3. Start Hot Standby Redundancy by entering:

```
redundancy control start
```

### Verify Hot Standby Redundancy Status After Starting

Verify the following on both sites after starting Hot Standby Redundancy to ensure that no further issues arise.

Warning:

- Contact the technical support team if there are any issues other than those described in this section
- Do not attempt to perform troubleshooting steps on the system yourself, as doing so could lead to further issues that might affect the system and the latest database

**At the Legacy Orchestrator CLI on the surviving site**

1. Open an SSH terminal session to the Legacy Orchestrator CLI on **the surviving site** using port 2200.
2. Log in as the **skylight** user.
3. Check Hot Standby Redundancy status by entering:

```
redundancy show status
```

**Expectation**:

- The "Global Status" should be "Started"
- The surviving site should be "Active" state and the replacement site should be "Passive" state
- The "Replication status" status should be "Synchronizing", and will change to "Up" once the synchronization is complete.

1. Check Hot Standby Redundancy configuration by entering:

```
redundancy show configuration
```

Expectation: "Preferred site" should be the surviving site (whether it's site-a or site-b depends on the actual circumstances of the system).

**At the Legacy Orchestrator CLI on the replacement site**

The steps and expectations are similar to "Verify at the Legacy Orchestrator CLI on the surviving site" section above.

**At the Docker host on the surviving site**

1. Open an SSH terminal session to **the surviving site** using port 22.
2. Log in as the **skylight** user.
3. Check drbd0 partition status by entering:

```
lsblk
```

**Expectation**: the 'drbd0' partition will be configured below the replicated partition and a mount point associated with 'mysql-ha'

**Example**:

```
$lsblk
...
sdb       8:16   0   30G  0 disk 
└─drbd0 147:0    0   30G  0 disk /home/skylight/24.09.1-GA/mysql-ha
```

**At the Docker host on the replacement site**

1. Open an SSH terminal session to **the replacement site** using port 22.
2. Log in as the **skylight** user.
3. Check drbd0 partition status by entering:

```
lsblk
```

**Expectation**: the 'drbd0' partition will be configured below the replicated partition and there is no mount point associated with 'mysql-ha'

**Example**:

```
$lsblk
...
sdb       8:16   0   30G  0 disk 
└─drbd0 147:0    0   30G  0 disk
```

### Update the Preferred Site to Its Original Configuration After Starting Hot Standby Redundancy

This step is optional. After starting Hot Standby Redundancy, follow this step to update the preferred site to its original state.

1. Open an SSH terminal session to the Legacy Orchestrator CLI on **the surviving site** using port 2200.
2. Log in as the **skylight** user.
3. Check the Hot Standby Redundancy status by entering:

```
redundancy show status
```

**Expectation**: The "Replication status" must be "Up". If it still shows "Synchronizing", wait until it changes to "Up".
4. Update the preferred site to its original configuration by entering:

```
redundancy control set-preferred-site {site-a | site-b}
```

**© 2026 Cisco and/or its affiliates. All rights reserved.** For more information about trademarks, please visit: [Cisco trademarks](https://www.cisco.com/c/en/us/about/legal/trademarks.html)[](https://www.cisco.com/c/en/us/about/legal/trademarks.html) For more information about legal terms, please visit: [Cisco legal terms](https://www.cisco.com/c/en/us/about/legal/terms-conditions.html)