--- title: "On-Prem Troubleshooting" slug: "on-prem-troubleshooting-runbook" updated: 2026-11-17T16:22:29Z published: 2026-11-17T16:22:29Z canonical: "docs.crossworkassurance.cisco.com/on-prem-troubleshooting-runbook" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.crossworkassurance.cisco.com/llms.txt > Use this file to discover all available pages before exploring further. # On-Prem Troubleshooting This article is intended to assist Customer Experience Engineers, Network Engineers, Field Engineers to diagnose and resolve common Crosswork Assurance on-premise installation and operational issues. It's organized by symptom so you can quickly find the right troubleshooting steps based on what you're seeing. ### Troubleshooting Methodology Before diving into specific issues, follow this systematic approach. It will save you time and help you avoid chasing symptoms instead of root causes: 1. **Identify symptoms:** Determine exactly what the user is experiencing. 2. **Check platform health:** Run standard health verification commands. 3. **Isolate the layer:** Determine if the issue is Infrastructure, Service, or Application related. 4. **Collect logs:** Gather relevant diagnostic information from the affected components. 5. **Apply fix:** Follow runbook steps or escalate to the next tier. ### Pre-Flight Check Failures These issues occur during installation before Crosswork Assurance is fully deployed. They're usually related to missing prerequisites or environment configuration. #### Error: socat not found **Symptom:** Installation fails with `unable to do port forwarding: socat not found`. **Cause:** The `socat` package is missing from the Kubernetes nodes. **Resolution:** ``` # RHEL/CentOS sudo yum install -y socat # Ubuntu/Debian sudo apt-get install -y socat # Verify installation which socat ``` #### Error: IPv6/Dual-Stack Mismatch **Symptom:** Pre-flight checks fail indicating a dual-stack version is required. **Cause:** An IPv6 address was detected but a non-dual-stack deployer version was selected. **Resolution:** Download the correct dual-stack version (e.g., `25.7.250-105-dualstack`) and re-run the installation. #### Error: Insufficient Resources **Symptom:** Pre-flight checks fail with resource warnings. **Cause:** Node does not meet minimum requirements (32 vCPU, 64GB RAM minimum per node). **Resolution:** 1. Verify node resources: `kubectl describe node <node-name>` 2. Scale up VM resources or add additional nodes 3. Re-run pre-flight checks #### Error: RBAC Permission Denied **Symptom:** Installation fails with permission errors. **Cause:** User lacks cluster-admin permissions. **Resolution:** ``` # Grant cluster-admin to current user kubectl create clusterrolebinding cluster-admin-binding \ --clusterrole=cluster-admin \ --user=$(gcloud config get-value account) ``` ### Reset and Reinstall Issues These issues occur when uninstalling and reinstalling Crosswork Assurance. A failed or partial installation can leave behind state that prevents a clean reinstall. #### Reset Fails with Connection Refused on Port 6443 **Symptom:** Running `./provider-connectivity-assurance reset` fails because the Kubernetes API is not reachable: ``` connection refused on port 6443 ``` **Cause:** The Kubernetes API server is not running or reachable, so the reset command cannot communicate with the cluster. **Resolution:** Use the `--force` flag to reset without requiring a live cluster: ``` sudo ./provider-connectivity-assurance reset --force ``` **Note:** During a forced reset, etcd connection and cluster errors are printed as services are torn down. This is expected behavior. #### Install Fails After Reset (Tenant Already Exists) **Symptom:** Reinstallation fails with a landlord job error: ``` ERROR: failed to create tenant in couchdb: "Unable to create Tenant pca. A Tenant with this name already exists" (Status code: 409) ``` **Cause:** A previous installation partially succeeded — the tenant was created in CouchDB — but failed before completing. The installer encounters a conflict when it tries to recreate the tenant. **Resolution:** 1. Reset the installation: ``` sudo ./provider-connectivity-assurance reset --force ``` 2. Delete leftover state: ``` sudo rm -rf /var/lib/embedded-cluster ``` 3. Restart the k0s controller: ``` sudo systemctl restart k0scontroller ``` 4. Verify the service is running: ``` sudo systemctl status k0scontroller ``` 5. Reinstall Crosswork Assurance. ### Post-Reboot Cluster Recovery If the host VM restarts unexpectedly — for example, due to a datastore outage, ESXi failure, or power cycle — the cluster may appear healthy (sessions running, data flowing into Inventory, Ready state in the Admin Console) but remain partially degraded. Common symptoms include one or more pods showing **Unavailable** in the Admin Console (for example, `sm-bizn1` or `sm-nbin1`). This runbook addresses the most common root causes after an unplanned VM reboot. Run the steps in order. #### Step 1: Restore NTP Synchronization A VM reboot can desynchronize the system clock, causing service failures across the cluster. ``` ssh root@ sudo timedatectl set-ntp true sudo systemctl enable --now systemd-timesyncd timedatectl status ``` Verify that `NTP synchronized: yes` appears in the output before continuing. #### Step 2: Free Disk Space Stale container images and accumulated logs can exceed safe thresholds after a reboot. Target disk usage below 75%. ``` # Remove unused container images sudo crictl rmi --prune # Remove journal logs older than 7 days sudo journalctl --vacuum-time=7d # Remove compressed log files older than 7 days sudo find /var/log -name "*.gz" -mtime +7 -delete ``` Verify with `df -h`. #### Step 3: Delete Stale Embedded-Cluster Lease An unclean shutdown can leave a stale lease in the `embedded-cluster` namespace. ``` kubectl delete lease 3f2343ef.replicated.com -n embedded-cluster ``` #### Step 4: Remove Evicted Pods Kubernetes marks pods as Evicted when disk or memory pressure occurs. Evicted pods typically appear in the `kotsadm` and `embedded-cluster` namespaces. Delete them to allow rescheduling. ``` # Delete all failed/evicted pods across all namespaces kubectl get pods --all-namespaces --field-selector=status.phase=Failed -o json | \ kubectl delete -f - ``` To delete individual pods by name: ``` kubectl delete pod -n --force --grace-period=0 ``` #### Step 5: Restart Unavailable Application Pods If specific application pods remain unavailable after the steps above, force-delete them to let the deployment controller reschedule them. ``` POD_NAME=$(kubectl get pods -n pca -l app=sm-bizn1 -o jsonpath='{.items[0].metadata.name}') kubectl delete pod $POD_NAME -n pca --force --grace-period=0 ``` Repeat for any other pods showing **Unavailable** in the Admin Console. #### Verification ``` # Confirm all pods are Running or Completed kubectl get pods -n pca # Confirm node status kubectl get nodes # Confirm NTP timedatectl status ``` The Admin Console should show all services in a **Ready** state with data flowing into Inventory and Sessions. ### Pod Issues Pod problems are the most common issues you'll encounter. This section covers the main pod states that indicate problems and how to resolve them. #### Pods Stuck in Pending **Diagnosis:** ``` kubectl describe pod -n pca ``` **Common Causes and Fixes:** The following table maps pending pod causes to their resolutions. Check the Events section of `kubectl describe pod` to identify which cause applies: | Cause | Fix | | --- | --- | | Insufficient CPU/Memory | Scale up nodes or reduce resource requests | | PVC not bound | Check StorageClass availability | | Node selector mismatch | Verify node labels | | Taints preventing scheduling | Add tolerations or remove taints | **Check PVC status:** ``` kubectl get pvc -n pca kubectl describe pvc -n pca ``` #### Pods in CrashLoopBackOff This means the pod is starting, crashing, and restarting repeatedly. The key is to check the logs from the previous container instance. **Diagnosis:** ``` # Get current logs kubectl logs -n pca # Get logs from previous crash kubectl logs -n pca --previous # Check events kubectl describe pod -n pca ``` **Common Causes:** - Missing configuration or secrets - Database connection failures - Out of memory (OOMKilled) - Application startup errors #### Pods in ImagePullBackOff **Symptom:** Pod can't pull container image. **Diagnosis:** ``` kubectl describe pod -n pca | grep -A 5 "Events" ``` **Common Causes:** - Invalid image pull secret - Network connectivity to registry - Image doesn't exist ### Database Issues Database problems affect multiple services. If you're seeing widespread failures, check the databases first. #### PostgreSQL Connection Failures **Symptoms:** Multiple services failing, authentication errors, configuration not loading. **Diagnosis:** ``` # Check PostgreSQL pod kubectl get pods -n pca -l app=postgres kubectl logs -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') # Test connectivity kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -c "SELECT 1;" ``` **Common Fixes:** 1. Check PVC is bound and has space 2. Verify PostgreSQL pod has enough memory 3. Check for connection limit exhaustion #### Druid Issues **Symptoms:** No data in dashboards, slow queries, historical data missing. **Diagnosis:** ``` # Check all Druid components kubectl get pods -n pca | grep druid # Check coordinator status kubectl port-forward -n pca svc/druid-coordinator 8081:8081 & curl http://localhost:8081/status # Check supervisor status curl http://localhost:8081/druid/indexer/v1/supervisor ``` **Common Fixes:** 1. Restart failed supervisors 2. Check MinIO connectivity for deep storage 3. Verify Kafka topics exist and have data ### Kafka Issues Kafka is the backbone of the data pipeline. If Kafka is unhealthy, no new data will flow through the system. #### High Consumer Lag **Symptoms:** Data delays, dashboards showing stale data. **Diagnosis:** ``` kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups ``` **Common Fixes:** 1. Check consumer pod health (Ignite, Druid indexers) 2. Verify consumer pods have enough resources 3. Scale up consumer replicas if needed #### Kafka Broker Not Ready **Diagnosis:** ``` # Check Kafka pods kubectl get pods -n pca -l app=kafka # Check Kafka logs kubectl logs -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') ``` **Common Causes:** - Zookeeper connectivity issues - Disk space exhaustion - Memory pressure ### Certificate Issues Certificate problems cause secure connections to fail. They can affect both internal service communication and external access. #### Certificate Expired or Invalid **Symptoms:** TLS errors, services can't communicate, browser certificate warnings. **Diagnosis:** ``` # Check certificate status kubectl get certificates -n pca # Check specific certificate kubectl describe certificate -n pca # Check expiration kubectl get secret -n pca nginx-tls-secret-cm -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate ``` **Common Fixes:** 1. Check cert-manager is running 2. Verify ClusterIssuer is configured correctly 3. Delete certificate to trigger renewal ### Network Issues Network problems can manifest in many ways. Use these checks when you suspect connectivity issues. #### Services Can't Communicate **Diagnosis:** ``` # Test DNS resolution kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- nslookup gather # Test service connectivity kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- curl -v http://gather/health ``` #### External Access Not Working **Diagnosis:** ``` # Check ingress kubectl get ingress -n pca # Check nginx pod kubectl get pods -n pca -l app=nginx kubectl logs -n pca $(kubectl get pods -n pca -l app=nginx -o jsonpath='{.items[0].metadata.name}') ``` ### Authentication Issues Authentication problems prevent users from logging in. Zitadel is the identity provider for PCA. #### Users Can't Log In **Diagnosis:** ``` # Check Zitadel pod kubectl get pods -n pca -l app=zitadel kubectl logs -n pca $(kubectl get pods -n pca -l app=zitadel -o jsonpath='{.items[0].metadata.name}') ``` **Common Fixes:** 1. Verify Zitadel database connectivity 2. Check SMTP configuration for password reset 3. Verify domain configuration matches DNS #### Default Admin Password Not Working **For Crosswork Assurance 25.07 CA and later:** - Password is displayed in Admin Console - Check Admin Console dashboard for credentials **Password Reset:** ``` # Access Zitadel UI # DNS mode: https://auth.{domain-name} # No-DNS mode: https://{external-ip}:3443 ``` #### Browser Displays Loading Screen **Symptoms:** The Crosswork Assurance loading screen progress bar stops partway through. The browser console shows a failed `feature-flags` request or a "Not authorized" error. This can occur on Chrome, Edge, and other browsers. **Cause:** The browser has not accepted the Zitadel authentication service certificate, causing the session to fail silently on the login redirect. **Resolution:** 1. Open a new browser tab and navigate to the Zitadel authentication page: - DNS mode: `https://auth.<domain-name>` - No-DNS mode: `https://<external-ip>:3443` 2. If prompted with a certificate warning, accept it to proceed. You do not need to log in to Zitadel. 3. Return to the Crosswork Assurance URL and reload the page. If the issue persists, clear your browser cookies or cache for the current browsing session and try again. ### Performance Issues Performance problems are often related to resource constraints or cache warming. #### Slow Queries **Symptoms:** UI dashboards load slowly, API timeouts. **Diagnosis:** ``` # Check Druid historical cache kubectl port-forward -n pca svc/health-clinic 8080:80 & curl http://localhost:8080/metrics | grep druid_historical_cache # Check Druid query metrics curl http://localhost:8080/metrics | grep druid_query ``` **Common Fixes:** 1. Wait for historical cache warm-up after restart 2. Check Druid resource allocation 3. Verify data retention policies #### High Memory Usage **Diagnosis:** ``` # Check pod resource usage kubectl top pods -n pca --sort-by=memory # Check node resources kubectl top nodes ``` ### Log Collection When you need to escalate or investigate further, collect these logs: #### Collecting Service Logs ``` # Single service logs kubectl logs -n pca > -logs.txt # All pods in namespace for pod in $(kubectl get pods -n pca -o jsonpath='{.items[*].metadata.name}'); do kubectl logs -n pca $pod > logs/$pod.txt 2>&1 done ``` #### Creating a Support Bundle The Admin Console can generate a comprehensive support bundle that includes logs, configuration, and diagnostic information: 1. Access the Admin Console. 2. Navigate to the **Troubleshoot** section. 3. Click **Analyze** to generate a comprehensive support bundle. 4. Download the bundle and share it with Cisco support. #### Collecting Events ``` kubectl get events -n pca --sort-by='.lastTimestamp' > pca-events.txt ``` ### Escalation Checklist Before escalating to Cisco support, ensure you have collected the following information. This will help support resolve your issue faster: - Support bundle from Admin Console - `helmStderr` logs from failed deployment - Output of `kubectl get pods -n pca -o wide` - Output of `kubectl get events -n pca` - Relevant service logs - Crosswork Assurance version and deployer version - Deployment type (VM/K8s, single/multi-node) - Network configuration (DNS/No-DNS, IPv4/IPv6) ### Quick Reference Commands Keep these commands handy for rapid troubleshooting: ``` # Overall health kubectl get pods -n pca kubectl get deployments -n pca kubectl get statefulsets -n pca # Logs kubectl logs -n pca kubectl logs -n pca --previous # Describe resources kubectl describe pod -n pca kubectl describe pvc -n pca # Events kubectl get events -n pca --sort-by='.lastTimestamp' # Resource usage kubectl top pods -n pca kubectl top nodes # Exec into pod kubectl exec -it -n pca -- /bin/sh ``` **© 2026 Cisco and/or its affiliates. All rights reserved.** For more information about trademarks, please visit: [Cisco trademarks](https://www.cisco.com/c/en/us/about/legal/trademarks.html)[](https://www.cisco.com/c/en/us/about/legal/trademarks.html) For more information about legal terms, please visit: [Cisco legal terms](https://www.cisco.com/c/en/us/about/legal/terms-conditions.html)