Replicated Cluster with H/W Sensor Management Worker Deployment

This article explains how to install Crosswork Assurance on virtual machines as a replicated system: a Replicated Embedded Cluster (Kubernetes, k0s) that runs Crosswork Assurance, plus one (or two) worker node(s) that runs H/W Sensor Management (H/W SM) in Docker.

Deployment Prerequisites

To install Crosswork Assurance, the Controller node must meet the following requirements:

• Linux operating system
  • Testing is performed on RHEL9, Ubuntu 22.04/24.04, and Debian 12
• x86-64 architecture
• systemd

The H/W SM worker node has its own requirements. See H/W Sensor Management Worker Node Requirements.

Computing Resources Plan

1. The deployment topology (use case)

• If the plan does not use the H/W SM (SSC session), deploy one VM for the Controller node; the H/W SM is not deployed by default.
• If the plan uses the H/W SM (SSC session), the requirement depends on whether the plan uses high availability feature (Warm Standby or Hot Standby) for the H/W SM worker node:
  • If non-high-availability for the H/W SM worker, deploy one additional VM for the H/W SM worker node
  • If high availability (Warm Standby or Hot Standby) for the H/W SM worker, deploy two additional VMs for the H/W SM worker nodes

2. Resource planning for a VM instance

The Controller requires one VM with the default lab-size resources. The H/W SM worker node uses the same CPU and RAM; only disk size varies by session scale: 750 GB for up to 50,000 sessions, 850 GB for up to 60,000 sessions, and 2 TB for up to 125,000 sessions.

2.1 Base resources

2.2 Total resources to provision with Sensor Collector add-on

Sensor Collector can be deployed on any VM. There are two types, S/W Sensor Collector and H/W Sensor Collector, and both need the same resources.

• If a Sensor Collector shares the Worker node, add 4 vCPU and 8 GB of memory per instance to that node's base resources.
• If both types share the same node, add 8 vCPU and 16 GB.

3. Disk layout

The following table details the mount point sizes for the Controller node.

The following table details the mount point sizes for the H/W SM worker node.

VM Networking Interface Requirements

Note: The resource requirements above assume hyperthreading. We recommend not leveraging hypervisor oversubscription, as the solution runs several latency-sensitive processes. If needed, try not to use a ratio of more than 4:1 vCPU to physical, and 20% over-commit for RAM.

Ports Required for Internal Exposure on Controller Node

By default, the Admin Console uses port 30000/TCP.

Ports Required for Bidirectional Communication Between Nodes

H/W Sensor Management Worker Node Requirements

Before deploying H/W Sensor Management from the Admin Console, make sure every target worker node meets the requirements below.

The H/W SM Worker Node Requirements

The components listed below constitute the reference environment used for development and validation. While other versions may be compatible, the versions specified here have been strictly verified for product functionality.

IPv6 Connectivity and Disabling the containerd Snapshotter Feature in Docker

The containerd image store enabled by default on Docker Engine can increase disk usage on the H/W SM worker node. To avoid unnecessary storage consumption, keep this feature disabled unless it is specifically required.

Add or update the following flags to the Docker service configuration file /etc/docker/daemon.json to allow the H/W SM to connect to IPv6 devices and disable the containerd snapshotter feature:

{
  "ip6tables": true,
  "experimental": true,
  "features": {
    "containerd-snapshotter": false
  }
}

Then restart the Docker daemon to apply the changes:

sudo systemctl daemon-reload
sudo systemctl restart docker

Local skylight User

Each host must have a local skylight user created before deploying. The account must meet all of the following requirements.

External URL Access

The following URLs must be accessible for installing Crosswork Assurance:

• proxy-registry.pca.cisco.com
• update.pca.cisco.com

To view maps in an airgap deployment, provide external access to the following Maptiler service URLs from the client machines you use to access Crosswork Assurance in a web browser:

• https://maptiler.com/
• https://api.maptiler.com/
• https://cloud.maptiler.com/
• https://docs.maptiler.com/

DNS Resolver Configuration

The host system must have a valid /etc/resolv.conf file with at least one nameserver entry configured. This is required even in airgap environments where external DNS resolution is not needed.

Airgap environments: The nameserver does not need to resolve external addresses, but the /etc/resolv.conf file must exist and contain a valid nameserver entry. You can configure a local or internal DNS server, or use a placeholder entry if no DNS services are available.

Example minimal configuration:

nameserver 127.0.0.1

DNS Configuration

Refer to Installation on Virtual Machines — DNS Configuration.

No-DNS Configuration

Refer to Installation on Virtual Machines — No-DNS Configuration.

(Optional) Dual-Stack Linux Infrastructure for IPv6 Support

To use IPv6 functionality with Crosswork Assurance, your underlying Linux infrastructure must be configured for dual-stack networking. Single-stack IPv6 is not supported.

If using IPv6, you must have DNS configured for your host machine. You will be asked to provide the server's domain name when deploying Crosswork Assurance from the Admin Console. The Admin Console must be accessed via the machine's IPv4 address.

For the H/W SM worker, dual-stack is a separate setting: Enable Dual-Stack in Host Information.

Increase Limits for Monitoring File System Events

Refer to Installation on Virtual Machines — Increase Limits for Monitoring File System Events.

Getting Started

IMPORTANT: Before you can access the Crosswork Assurance software for download (for either a proof-of-concept demo or a customer production environment), an order must be processed through the Cisco Sales process. Contact your Account team to process your order.

Upon order completion, Cisco creates your software portal download account, and you can download Crosswork Assurance from the Download Portal.

You must provide a valid email to access the Download Portal. You may be emailed a temporary access code to continue.

In the Download Portal, follow the Installation Guide to obtain the tar archive for the provider-connectivity-assurance installation asset. First, prepare an instance by giving it a name and selecting whether it can make outbound requests to Cisco directly, needs an HTTPS proxy, or will be an airgap environment.

Order of work for this deployment:

1. Prepare the Controller VM node and the H/W SM Worker VM node(s)
2. Install the Crosswork Assurance Admin Console
3. Fill in all configuration, including the H/W SM, before the first save
4. Deploy. A single deploy brings up Crosswork Assurance, joins the H/W SM worker, and installs the H/W SM worker.

Online Installer Download

Refer to Installation on Virtual Machines — Online Installer Download.

Airgap Installer Download

Refer to Installation on Virtual Machines — Airgap Installer Download.

Version Selection

Refer to Installation on Virtual Machines — Version Selection.

Installation

Refer to Installation on Virtual Machines — Installation.

Access the Crosswork Assurance Admin Console

Refer to Installation on Virtual Machines — Access the Crosswork Assurance Admin Console.

Adding Nodes to the Cluster

Do not use this procedure for the H/W SM worker node. The H/W SM worker is joined to the cluster automatically during deployment, using the details in Configuring the H/W SM. Use Add node only for additional Kubernetes nodes that run Crosswork Assurance.

Refer to Installation on Virtual Machines — Adding Nodes to the Cluster.

Removing Nodes from the Cluster

To reset the H/W SM worker node, do not use this procedure. Use Resetting the H/W SM Worker Node instead.

Refer to Installation on Virtual Machines — Removing Nodes from the Cluster.

Configuring Crosswork Assurance

Follow the steps below to configure your Crosswork Assurance deployment. Each configuration option is documented with details on its purpose. Most options have default values; use the default values when possible.

Hybrid On-Prem / Cloud Split

Hybrid mode connects this on-prem Crosswork Assurance deployment to a cloud Crosswork Assurance tenant and splits the work between them:

• Runs on-prem: Sensor orchestration, YANG gateway, and Sensor Management
• Runs in the cloud: Gather (MOs), analytics pipeline, gNMI streaming, and alerts

Hybrid mode is optional and is disabled by default. It is separate from the H/W SM worker: you can deploy the H/W SM worker with hybrid mode disabled. Enable it only when this deployment is paired with a cloud Crosswork Assurance tenant.

To enable it, go to Config → Hybrid On-Prem / Cloud Split and fill in the fields below. Get the values from the cloud Crosswork Assurance tenant. Turn on Show how to get these values to see the steps in the Admin Console.

The required fields apply only when hybrid mode is enabled. As with the other groups, complete this group before you save the configuration for the first time.

Configuring H/W Sensor Management

Do not join the worker or install SM Docker by hand. The H/W SM worker is joined to the cluster automatically during deployment.

Depending on whether the plan uses the high availability feature (Warm Standby or Hot Standby) for the H/W SM worker node, follow the corresponding article for details:

• Non-High-Availability: Configuring H/W Sensor Management - Non-High-Availability System
• Warm Standby: Configuring H/W Sensor Management - Warm Standby System
• Hot Standby: Configuring H/W Sensor Management - Hot Standby System

Resetting the H/W SM Worker Node

See Resetting the H/W SM Worker Node for details.

SMTP Support

Refer to Installation on Virtual Machines — SMTP Support.

Pre-Flight Checks

Refer to Installation on Virtual Machines — Pre-Flight Checks.

First-Time Login to H/W Sensor Management UI

Once the deployment is complete, log in for the first time using the default admin user for the deployment.

Note: The default admin user's username comes from the deployment name and domain name entered in the Admin Console configuration.

As the default admin user, you can log in to the web UI hosted at:

Note: On first login, change the default admin user's password.

Accessing the Identity and Access Management UI

Identity and access management in Crosswork Assurance is implemented with Zitadel. Use the Zitadel UI to create service users for API integration and to configure SSO for the deployment.

Configuring TLS Clients with Crosswork Assurance

Multiple clients that connect to Crosswork Assurance require a TLS connection. For example:

• Sensor Collectors
• MQTT Clients
• Web browser

If you use the self-signed certificate option in the Admin Console, export the CA to a file and add it to your trust stores:

kubectl get secret -n pca nginx-tls-secret-cm -o jsonpath={.data."ca\.crt"} \
| base64 -d > ca.crt

This command and additional details are also listed in the Admin Console under the Certificates section when the generate option is selected.

Using Custom Certificates (Bring Your Own Certificate)

If you provide your own TLS certificates instead of the Crosswork Assurance-generated self-signed option, your certificates must be signed by a trusted Certificate Authority (CA).

Important: Crosswork Assurance enforces certificate validation in compliance with Cisco security requirements. Self-signed certificates that are not issued by a trusted CA are rejected and cause connection failures.

Not supported: self-signed certificates created outside of Crosswork Assurance that are not signed by a CA in the system trust store. If you require custom self-signed certificates, contact Cisco support.

Upgrading to a New Deployer Version

Crosswork Assurance can be upgraded to a new deployer to pick up patch releases and major releases. Upgrades are done in the Admin Console at https://{machine-ip}:30000.

When a new version is available, it appears on the Admin Console Dashboard under Latest Available Update. Click Go to Version history, then click Deploy next to the version you want. Follow the steps in the Deployment view: update the configuration, run the pre-flight checks, and confirm the deployment.

The H/W SM version is set separately by Application Version in Application Configuration. Changing it and redeploying is expected to upgrade SM on the worker.

Upgrading an Airgap Deployment

1. Log in to the software portal and go to the Update tab to check for and download the new airgap bundle.
2. Find your airgap instance record. If you have not created an instance record to track your install, you can create it now; you must specify the deployer version you are currently running, which is shown in the Admin Console of your airgap instance.
3. If an update is available, this view shows it. Click it to see how to apply it.
4. Run the curl command from your local terminal to download the airgap bundle.
5. Transfer the bundle to the primary node of your deployment (through VPN or another method).
6. Extract and run the provider-connectivity-assurance script in update mode.
7. Open the Admin Console of your deployment. The new update is available to apply, as described above.