✨ New: Try our AI‑powered Search (Ctrl + K) — Read more

On-Prem Troubleshooting

Prev Next

This article is intended to assist Customer Experience Engineers, Network Engineers, Field Engineers to diagnose and resolve common Crosswork Assurance on-premise installation and operational issues. It's organized by symptom so you can quickly find the right troubleshooting steps based on what you're seeing.

Troubleshooting Methodology

Before diving into specific issues, follow this systematic approach. It will save you time and help you avoid chasing symptoms instead of root causes:

  1. Identify symptoms: Determine exactly what the user is experiencing.
  2. Check platform health: Run standard health verification commands.
  3. Isolate the layer: Determine if the issue is Infrastructure, Service, or Application related.
  4. Collect logs: Gather relevant diagnostic information from the affected components.
  5. Apply fix: Follow runbook steps or escalate to the next tier.

Pre-Flight Check Failures

These issues occur during installation before Crosswork Assurance is fully deployed. They're usually related to missing prerequisites or environment configuration.

Error: socat not found

Symptom: Installation fails with unable to do port forwarding: socat not found.

Cause: The socat package is missing from the Kubernetes nodes.

Resolution:

# RHEL/CentOS
sudo yum install -y socat

# Ubuntu/Debian
sudo apt-get install -y socat

# Verify installation
which socat

Error: IPv6/Dual-Stack Mismatch

Symptom: Pre-flight checks fail indicating a dual-stack version is required.

Cause: An IPv6 address was detected but a non-dual-stack deployer version was selected.

Resolution: Download the correct dual-stack version (e.g., 25.7.250-105-dualstack) and re-run the installation.

Error: Insufficient Resources

Symptom: Pre-flight checks fail with resource warnings.

Cause: Node does not meet minimum requirements (32 vCPU, 64GB RAM minimum per node).

Resolution:

  1. Verify node resources: kubectl describe node <node-name>
  2. Scale up VM resources or add additional nodes
  3. Re-run pre-flight checks

Error: RBAC Permission Denied

Symptom: Installation fails with permission errors.

Cause: User lacks cluster-admin permissions.

Resolution:

# Grant cluster-admin to current user
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user=$(gcloud config get-value account)

Reset and Reinstall Issues

These issues occur when uninstalling and reinstalling Crosswork Assurance. A failed or partial installation can leave behind state that prevents a clean reinstall.

Reset Fails with Connection Refused on Port 6443

Symptom: Running ./provider-connectivity-assurance reset fails because the Kubernetes API is not reachable:

connection refused on port 6443

Cause: The Kubernetes API server is not running or reachable, so the reset command cannot communicate with the cluster.

Resolution: Use the --force flag to reset without requiring a live cluster:

sudo ./provider-connectivity-assurance reset --force

Note: During a forced reset, etcd connection and cluster errors are printed as services are torn down. This is expected behavior.

Install Fails After Reset (Tenant Already Exists)

Symptom: Reinstallation fails with a landlord job error:

ERROR: failed to create tenant in couchdb:
"Unable to create Tenant pca. A Tenant with this name already exists" (Status code: 409)

Cause: A previous installation partially succeeded — the tenant was created in CouchDB — but failed before completing. The installer encounters a conflict when it tries to recreate the tenant.

Resolution:

  1. Reset the installation:
    sudo ./provider-connectivity-assurance reset --force
    
  2. Delete leftover state:
    sudo rm -rf /var/lib/embedded-cluster
    
  3. Restart the k0s controller:
    sudo systemctl restart k0scontroller
    
  4. Verify the service is running:
    sudo systemctl status k0scontroller
    
  5. Reinstall Crosswork Assurance.

Post-Reboot Cluster Recovery

If the host VM restarts unexpectedly — for example, due to a datastore outage, ESXi failure, or power cycle — the cluster may appear healthy (sessions running, data flowing into Inventory, Ready state in the Admin Console) but remain partially degraded. Common symptoms include one or more pods showing Unavailable in the Admin Console (for example, sm-bizn1 or sm-nbin1).

This runbook addresses the most common root causes after an unplanned VM reboot. Run the steps in order.

Step 1: Restore NTP Synchronization

A VM reboot can desynchronize the system clock, causing service failures across the cluster.

ssh root@<node-ip>
sudo timedatectl set-ntp true
sudo systemctl enable --now systemd-timesyncd
timedatectl status

Verify that NTP synchronized: yes appears in the output before continuing.

Step 2: Free Disk Space

Stale container images and accumulated logs can exceed safe thresholds after a reboot. Target disk usage below 75%.

# Remove unused container images
sudo crictl rmi --prune

# Remove journal logs older than 7 days
sudo journalctl --vacuum-time=7d

# Remove compressed log files older than 7 days
sudo find /var/log -name "*.gz" -mtime +7 -delete

Verify with df -h.

Step 3: Delete Stale Embedded-Cluster Lease

An unclean shutdown can leave a stale lease in the embedded-cluster namespace.

kubectl delete lease 3f2343ef.replicated.com -n embedded-cluster

Step 4: Remove Evicted Pods

Kubernetes marks pods as Evicted when disk or memory pressure occurs. Evicted pods typically appear in the kotsadm and embedded-cluster namespaces. Delete them to allow rescheduling.

# Delete all failed/evicted pods across all namespaces
kubectl get pods --all-namespaces --field-selector=status.phase=Failed -o json | \
  kubectl delete -f -

To delete individual pods by name:

kubectl delete pod <pod-name> -n <namespace> --force --grace-period=0

Step 5: Restart Unavailable Application Pods

If specific application pods remain unavailable after the steps above, force-delete them to let the deployment controller reschedule them.

POD_NAME=$(kubectl get pods -n pca -l app=sm-bizn1 -o jsonpath='{.items[0].metadata.name}')
kubectl delete pod $POD_NAME -n pca --force --grace-period=0

Repeat for any other pods showing Unavailable in the Admin Console.

Verification

# Confirm all pods are Running or Completed
kubectl get pods -n pca

# Confirm node status
kubectl get nodes

# Confirm NTP
timedatectl status

The Admin Console should show all services in a Ready state with data flowing into Inventory and Sessions.

Pod Issues

Pod problems are the most common issues you'll encounter. This section covers the main pod states that indicate problems and how to resolve them.

Pods Stuck in Pending

Diagnosis:

kubectl describe pod <pod-name> -n pca

Common Causes and Fixes:

The following table maps pending pod causes to their resolutions. Check the Events section of kubectl describe pod to identify which cause applies:

Cause Fix
Insufficient CPU/Memory Scale up nodes or reduce resource requests
PVC not bound Check StorageClass availability
Node selector mismatch Verify node labels
Taints preventing scheduling Add tolerations or remove taints

Check PVC status:

kubectl get pvc -n pca
kubectl describe pvc <pvc-name> -n pca

Pods in CrashLoopBackOff

This means the pod is starting, crashing, and restarting repeatedly. The key is to check the logs from the previous container instance.

Diagnosis:

# Get current logs
kubectl logs <pod-name> -n pca

# Get logs from previous crash
kubectl logs <pod-name> -n pca --previous

# Check events
kubectl describe pod <pod-name> -n pca

Common Causes:

  • Missing configuration or secrets
  • Database connection failures
  • Out of memory (OOMKilled)
  • Application startup errors

Pods in ImagePullBackOff

Symptom: Pod can't pull container image.

Diagnosis:

kubectl describe pod <pod-name> -n pca | grep -A 5 "Events"

Common Causes:

  • Invalid image pull secret
  • Network connectivity to registry
  • Image doesn't exist

Database Issues

Database problems affect multiple services. If you're seeing widespread failures, check the databases first.

PostgreSQL Connection Failures

Symptoms: Multiple services failing, authentication errors, configuration not loading.

Diagnosis:

# Check PostgreSQL pod
kubectl get pods -n pca -l app=postgres
kubectl logs -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}')

# Test connectivity
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -c "SELECT 1;"

Common Fixes:

  1. Check PVC is bound and has space
  2. Verify PostgreSQL pod has enough memory
  3. Check for connection limit exhaustion

Druid Issues

Symptoms: No data in dashboards, slow queries, historical data missing.

Diagnosis:

# Check all Druid components
kubectl get pods -n pca | grep druid

# Check coordinator status
kubectl port-forward -n pca svc/druid-coordinator 8081:8081 &
curl http://localhost:8081/status

# Check supervisor status
curl http://localhost:8081/druid/indexer/v1/supervisor

Common Fixes:

  1. Restart failed supervisors
  2. Check MinIO connectivity for deep storage
  3. Verify Kafka topics exist and have data

Kafka Issues

Kafka is the backbone of the data pipeline. If Kafka is unhealthy, no new data will flow through the system.

High Consumer Lag

Symptoms: Data delays, dashboards showing stale data.

Diagnosis:

kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \
  kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups

Common Fixes:

  1. Check consumer pod health (Ignite, Druid indexers)
  2. Verify consumer pods have enough resources
  3. Scale up consumer replicas if needed

Kafka Broker Not Ready

Diagnosis:

# Check Kafka pods
kubectl get pods -n pca -l app=kafka

# Check Kafka logs
kubectl logs -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}')

Common Causes:

  • Zookeeper connectivity issues
  • Disk space exhaustion
  • Memory pressure

Certificate Issues

Certificate problems cause secure connections to fail. They can affect both internal service communication and external access.

Certificate Expired or Invalid

Symptoms: TLS errors, services can't communicate, browser certificate warnings.

Diagnosis:

# Check certificate status
kubectl get certificates -n pca

# Check specific certificate
kubectl describe certificate <cert-name> -n pca

# Check expiration
kubectl get secret -n pca nginx-tls-secret-cm -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

Common Fixes:

  1. Check cert-manager is running
  2. Verify ClusterIssuer is configured correctly
  3. Delete certificate to trigger renewal

Network Issues

Network problems can manifest in many ways. Use these checks when you suspect connectivity issues.

Services Can't Communicate

Diagnosis:

# Test DNS resolution
kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- nslookup gather

# Test service connectivity
kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- curl -v http://gather/health

External Access Not Working

Diagnosis:

# Check ingress
kubectl get ingress -n pca

# Check nginx pod
kubectl get pods -n pca -l app=nginx
kubectl logs -n pca $(kubectl get pods -n pca -l app=nginx -o jsonpath='{.items[0].metadata.name}')

Authentication Issues

Authentication problems prevent users from logging in. Zitadel is the identity provider for PCA.

Users Can't Log In

Diagnosis:

# Check Zitadel pod
kubectl get pods -n pca -l app=zitadel
kubectl logs -n pca $(kubectl get pods -n pca -l app=zitadel -o jsonpath='{.items[0].metadata.name}')

Common Fixes:

  1. Verify Zitadel database connectivity
  2. Check SMTP configuration for password reset
  3. Verify domain configuration matches DNS

Default Admin Password Not Working

For Crosswork Assurance 25.07 CA and later:

  • Password is displayed in Admin Console
  • Check Admin Console dashboard for credentials

Password Reset:

# Access Zitadel UI
# DNS mode: https://auth.{domain-name}
# No-DNS mode: https://{external-ip}:3443

Browser Displays Loading Screen

Symptoms: The Crosswork Assurance loading screen progress bar stops partway through. The browser console shows a failed feature-flags request or a "Not authorized" error. This can occur on Chrome, Edge, and other browsers.

Cause: The browser has not accepted the Zitadel authentication service certificate, causing the session to fail silently on the login redirect.

Resolution:

  1. Open a new browser tab and navigate to the Zitadel authentication page:
    • DNS mode: https://auth.<domain-name>
    • No-DNS mode: https://<external-ip>:3443
  2. If prompted with a certificate warning, accept it to proceed. You do not need to log in to Zitadel.
  3. Return to the Crosswork Assurance URL and reload the page.

If the issue persists, clear your browser cookies or cache for the current browsing session and try again.

Performance Issues

Performance problems are often related to resource constraints or cache warming.

Slow Queries

Symptoms: UI dashboards load slowly, API timeouts.

Diagnosis:

# Check Druid historical cache
kubectl port-forward -n pca svc/health-clinic 8080:80 &
curl http://localhost:8080/metrics | grep druid_historical_cache

# Check Druid query metrics
curl http://localhost:8080/metrics | grep druid_query

Common Fixes:

  1. Wait for historical cache warm-up after restart
  2. Check Druid resource allocation
  3. Verify data retention policies

High Memory Usage

Diagnosis:

# Check pod resource usage
kubectl top pods -n pca --sort-by=memory

# Check node resources
kubectl top nodes

Log Collection

When you need to escalate or investigate further, collect these logs:

Collecting Service Logs

# Single service logs
kubectl logs -n pca <pod-name> > <service>-logs.txt

# All pods in namespace
for pod in $(kubectl get pods -n pca -o jsonpath='{.items[*].metadata.name}'); do
  kubectl logs -n pca $pod > logs/$pod.txt 2>&1
done

Creating a Support Bundle

The Admin Console can generate a comprehensive support bundle that includes logs, configuration, and diagnostic information:

  1. Access the Admin Console.
  2. Navigate to the Troubleshoot section.
  3. Click Analyze to generate a comprehensive support bundle.
  4. Download the bundle and share it with Cisco support.

Collecting Events

kubectl get events -n pca --sort-by='.lastTimestamp' > pca-events.txt

Escalation Checklist

Before escalating to Cisco support, ensure you have collected the following information. This will help support resolve your issue faster:

  • Support bundle from Admin Console
  • helmStderr logs from failed deployment
  • Output of kubectl get pods -n pca -o wide
  • Output of kubectl get events -n pca
  • Relevant service logs
  • Crosswork Assurance version and deployer version
  • Deployment type (VM/K8s, single/multi-node)
  • Network configuration (DNS/No-DNS, IPv4/IPv6)

Quick Reference Commands

Keep these commands handy for rapid troubleshooting:

# Overall health
kubectl get pods -n pca
kubectl get deployments -n pca
kubectl get statefulsets -n pca

# Logs
kubectl logs -n pca <pod-name>
kubectl logs -n pca <pod-name> --previous

# Describe resources
kubectl describe pod -n pca <pod-name>
kubectl describe pvc -n pca <pvc-name>

# Events
kubectl get events -n pca --sort-by='.lastTimestamp'

# Resource usage
kubectl top pods -n pca
kubectl top nodes

# Exec into pod
kubectl exec -it -n pca <pod-name> -- /bin/sh

© 2026 Cisco and/or its affiliates. All rights reserved.

For more information about trademarks, please visit:
Cisco trademarks 
For more information about legal terms, please visit:
Cisco legal terms