Backup, Restore and Rollback Planning Guide

This guide ties together the per-datastore backup articles into a single plan for full-system backup, restore ordering, and rollback, including the air-gapped scenarios where you must move backup artifacts offline.

Keep in mind that backups are point-in-time snapshots. Any data written after the backup timestamp is lost and must be recreated.

Getting Started

You will need:

• Admin access to the Crosswork Assurance deployment (Kubernetes or Docker Swarm)
• Access to the object storage backend (MinIO for K8s, GCS for Swarm) used for backups
• Familiarity with your existing per-datastore backup articles:
  • Backup and Restore (Docker Swarm)
    • OpenSearch Backup and Restore (Docker Swarm)
  • Backup and Restore (Kubernetes)
    • PostgreSQL Backup and Restore
    • CouchDB Backup and Restore
    • Dgraph Backup and Restore
    • OpenSearch Backup and Restore

1. Backup Strategy

Automated Backup Schedule:

Retention:

2. Per-Datastore Backup

All datastores use Kubernetes CronJobs that write backups to the object storage backend.

Verify CronJobs are running:

kubectl get cronjobs -n pca

Expected output includes:

Dgraph backups are controlled by the stitchit-env-config ConfigMap (not a CronJob). Verify Dgraph backup is configured — values should not be empty:

kubectl describe configmap -n pca stitchit-env-config | grep BACKUP

Note: Some deployments may have empty or misconfigured Dgraph backup values. If the output is empty, Dgraph backups are not running.

Manually Triggering a Backup

From any pod with curl (e.g., airflow-0):

For Dgraph, either update the stitchit-env-config schedule temporarily, or use the GraphQL admin endpoint.

First, retrieve MinIO credentials:

kubectl get secret -n pca pca-minio -o jsonpath='{.data.rootUser}' | base64 -d
kubectl get secret -n pca pca-minio -o jsonpath='{.data.rootPassword}' | base64 -d

Then trigger the backup:

curl -H "Content-Type: application/graphql" -X POST dgraphalpha1-0:8080/admin -d '
mutation {
  export(input: {
    destination: "minio://pca-minio-hl.pca.svc.cluster.local:9000/<bucket>?secure=false"
    accessKey: "<MINIO_ROOT_USER>"
    secretKey: "<MINIO_ROOT_PASSWORD>"
  }) {
    response { message code }
  }
}'

3. Full-System Backup

There is no single coordinated snapshot mechanism. Back up each datastore individually using its CronJob or manual trigger.

Best practice: Take a backup of all datastores immediately before any planned activity (upgrade, migration, configuration change) to minimize potential data loss. Backup order does not matter — what matters is that all datastores are backed up before the activity begins.

4. Backup Transfer

This section describes how to copy backup files from MinIO to the VM filesystem. This process applies to all deployment types (air-gapped and non-air-gapped). The MinIO image does not include tar, so kubectl cp cannot be used directly — files must be streamed using cat.

4.1 Locate Backups in MinIO

kubectl exec -it -n pca pca-minio-pool-0-0 -- sh
. /tmp/minio/config.env
mc alias set --insecure pca https://localhost:9000 "$MINIO_ROOT_USER" "$MINIO_ROOT_PASSWORD"

Discover your environment's bucket name:

kubectl exec -n pca <opensearch-pod> -- printenv STORAGE_BUCKET

List available backups:

Example:

mc --insecure ls pca/embedded-pca-us-central1-db-backup-accedian-net/postgres/v2

4.2 Copy Backup to Pod Filesystem

mc --insecure cp -r pca/<bucket>/postgres/v2/postgresBackup-<timestamp> /tmp/

4.3 Stream to VM (MinIO tar Limitation)

Because MinIO lacks tar, stream files using cat:

# On VM
export PG_BKUP="postgresBackup-2026-02-06T01:00:01"
mkdir $PG_BKUP

kubectl exec -n pca pca-minio-pool-0-0 -- cat /tmp/$PG_BKUP/base.tar > $PG_BKUP/base.tar
kubectl exec -n pca pca-minio-pool-0-0 -- cat /tmp/$PG_BKUP/pg_wal.tar > $PG_BKUP/pg_wal.tar

4.4 Verify Checksums

kubectl exec -n pca pca-minio-pool-0-0 -- cksum /tmp/$PG_BKUP/base.tar
cksum $PG_BKUP/base.tar

Checksums must match exactly before transferring offline.

4.5 Transfer Offline

Transfer via approved media (USB, secure file transfer). Verify checksums on the destination side before restore.

5. Restore Procedure

Restore all datastores that were backed up. Order does not matter as long as all datastores are restored before restarting services.

• PostgreSQL
• Yang PostgreSQL
• CouchDB
• Dgraph
• OpenSearch

Note: For complex recovery scenarios or if you encounter issues during restoration, contact Cisco TAC for assistance.

Pre-Restore Checklist

Before starting a restore, verify:

• Backup files transferred to target node and checksums verified
• Sufficient disk space on target node(s)
• Current data directories backed up (in case restore fails)
• Dependent services identified for shutdown

General Restore Pattern (Kubernetes)

Each datastore follows a similar pattern:

1. Stop dependent services — scale deployments/statefulsets to 0
2. Stop the datastore — scale the datastore statefulset to 0
3. Replace data directory — on the OpenEBS host node, replace the PVC contents with backup
4. Start the datastore — scale back to 1, verify logs
5. Start dependent services — scale deployments back to 1

OpenEBS Storage Location

Embedded/air-gapped deployments use OpenEBS local storage:

kubectl get storageclass
# Expected: openebs-hostpath (default)

PVC data resides at: /var/lib/embedded-cluster/openebs-local/<pvc-id>/

To find the exact path for a specific PVC (e.g., PostgreSQL):

kubectl get pv $(kubectl get pvc -n pca postgres-data-postgres-0 -o jsonpath='{.spec.volumeName}') \
  -o jsonpath='{.spec.local.path}'

Per-Datastore Restore References

• PostgreSQL → PostgreSQL Backup and Restore
• Yang PostgreSQL → PostgreSQL Backup and Restore (same procedure)
• CouchDB → CouchDB Backup and Restore
• OpenSearch → OpenSearch Backup and Restore
• Dgraph → Dgraph Backup and Restore

6. Rollback Planning

Before performing operations that may require a rollback (upgrades, migrations, configuration changes):

1. Take a fresh backup of all datastores. Record the backup archive path and timestamp.
2. Define rollback triggers — what symptoms trigger rollback? Examples:
   • Service endpoints unresponsive after change
   • Data validation failures in logs
   • User-visible errors or dashboard rendering failures
3. Perform the change.
4. Validate. If rollback triggers trip, restore per section 5.

7. Restore Validation Checklist

After any restoration, verify:

• All service endpoints respond (test via kubectl get pods -n pca — all Running)
• PostgreSQL connections work: kubectl exec -it -n pca postgres-0 -- psql -U postgres -c "SELECT 1"
• CouchDB UI accessible
• Alert history and active alerts match the backup timestamp
• Dashboards render with expected data ranges
• No cross-datastore consistency errors in logs

Post-Restore Data Backfill

Note: Any activities performed since the backup timestamp must be re-run after restoration.

Backups are taken at 01:00 UTC (04:00 UTC for Dgraph). Any data generated after that time (typically via Airflow jobs) will be missing. Re-run ingestion jobs manually if needed:

for hour in 05 06 07 08 09 10 11 12 13 14 15 16 17; do
  payload=$(printf '{"logical_date":"2026-02-05T%s:00:00Z"}' "$hour")
  curl -X POST http://airflow:8080/airflowui/api/v1/dags/Batch_Ingestion_v1.1_contemporary/dagRuns \
    -H 'Content-Type: application/json' \
    -d "$payload"
  sleep 180  # Only 2 Airflow job slots available
done