This article is intended to assist Customer Experience Engineers, Network Engineers, Field Engineers to diagnose and resolve common Crosswork Assurance on-premise installation and operational issues. It's organized by symptom so you can quickly find the right troubleshooting steps based on what you're seeing.
Troubleshooting Methodology
Before diving into specific issues, follow this systematic approach. It will save you time and help you avoid chasing symptoms instead of root causes:
- Identify symptoms: Determine exactly what the user is experiencing.
- Check platform health: Run standard health verification commands.
- Isolate the layer: Determine if the issue is Infrastructure, Service, or Application related.
- Collect logs: Gather relevant diagnostic information from the affected components.
- Apply fix: Follow runbook steps or escalate to the next tier.
Pre-Flight Check Failures
These issues occur during installation before Crosswork Assurance is fully deployed. They're usually related to missing prerequisites or environment configuration.
Error: socat not found
Symptom: Installation fails with unable to do port forwarding: socat not found.
Cause: The socat package is missing from the Kubernetes nodes.
Resolution:
# RHEL/CentOS
sudo yum install -y socat
# Ubuntu/Debian
sudo apt-get install -y socat
# Verify installation
which socat
Error: IPv6/Dual-Stack Mismatch
Symptom: Pre-flight checks fail indicating a dual-stack version is required.
Cause: An IPv6 address was detected but a non-dual-stack deployer version was selected.
Resolution: Download the correct dual-stack version (e.g., 25.7.250-105-dualstack) and re-run the installation.
Error: Insufficient Resources
Symptom: Pre-flight checks fail with resource warnings.
Cause: Node does not meet minimum requirements (32 vCPU, 64GB RAM minimum per node).
Resolution:
- Verify node resources:
kubectl describe node <node-name> - Scale up VM resources or add additional nodes
- Re-run pre-flight checks
Error: RBAC Permission Denied
Symptom: Installation fails with permission errors.
Cause: User lacks cluster-admin permissions.
Resolution:
# Grant cluster-admin to current user
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole=cluster-admin \
--user=$(gcloud config get-value account)
Reset and Reinstall Issues
These issues occur when uninstalling and reinstalling Crosswork Assurance. A failed or partial installation can leave behind state that prevents a clean reinstall.
Reset Fails with Connection Refused on Port 6443
Symptom: Running ./provider-connectivity-assurance reset fails because the Kubernetes API is not reachable:
connection refused on port 6443
Cause: The Kubernetes API server is not running or reachable, so the reset command cannot communicate with the cluster.
Resolution: Use the --force flag to reset without requiring a live cluster:
sudo ./provider-connectivity-assurance reset --force
Note: During a forced reset, etcd connection and cluster errors are printed as services are torn down. This is expected behavior.
Install Fails After Reset (Tenant Already Exists)
Symptom: Reinstallation fails with a landlord job error:
ERROR: failed to create tenant in couchdb:
"Unable to create Tenant pca. A Tenant with this name already exists" (Status code: 409)
Cause: A previous installation partially succeeded — the tenant was created in CouchDB — but failed before completing. The installer encounters a conflict when it tries to recreate the tenant.
Resolution:
- Reset the installation:
sudo ./provider-connectivity-assurance reset --force - Delete leftover state:
sudo rm -rf /var/lib/embedded-cluster - Restart the k0s controller:
sudo systemctl restart k0scontroller - Verify the service is running:
sudo systemctl status k0scontroller - Reinstall Crosswork Assurance.
Post-Reboot Cluster Recovery
If the host VM restarts unexpectedly — for example, due to a datastore outage, ESXi failure, or power cycle — the cluster may appear healthy (sessions running, data flowing into Inventory, Ready state in the Admin Console) but remain partially degraded. Common symptoms include one or more pods showing Unavailable in the Admin Console (for example, sm-bizn1 or sm-nbin1).
This runbook addresses the most common root causes after an unplanned VM reboot. Run the steps in order.
Step 1: Restore NTP Synchronization
A VM reboot can desynchronize the system clock, causing service failures across the cluster.
ssh root@<node-ip>
sudo timedatectl set-ntp true
sudo systemctl enable --now systemd-timesyncd
timedatectl status
Verify that NTP synchronized: yes appears in the output before continuing.
Step 2: Free Disk Space
Stale container images and accumulated logs can exceed safe thresholds after a reboot. Target disk usage below 75%.
# Remove unused container images
sudo crictl rmi --prune
# Remove journal logs older than 7 days
sudo journalctl --vacuum-time=7d
# Remove compressed log files older than 7 days
sudo find /var/log -name "*.gz" -mtime +7 -delete
Verify with df -h.
Step 3: Delete Stale Embedded-Cluster Lease
An unclean shutdown can leave a stale lease in the embedded-cluster namespace.
kubectl delete lease 3f2343ef.replicated.com -n embedded-cluster
Step 4: Remove Evicted Pods
Kubernetes marks pods as Evicted when disk or memory pressure occurs. Evicted pods typically appear in the kotsadm and embedded-cluster namespaces. Delete them to allow rescheduling.
# Delete all failed/evicted pods across all namespaces
kubectl get pods --all-namespaces --field-selector=status.phase=Failed -o json | \
kubectl delete -f -
To delete individual pods by name:
kubectl delete pod <pod-name> -n <namespace> --force --grace-period=0
Step 5: Restart Unavailable Application Pods
If specific application pods remain unavailable after the steps above, force-delete them to let the deployment controller reschedule them.
POD_NAME=$(kubectl get pods -n pca -l app=sm-bizn1 -o jsonpath='{.items[0].metadata.name}')
kubectl delete pod $POD_NAME -n pca --force --grace-period=0
Repeat for any other pods showing Unavailable in the Admin Console.
Verification
# Confirm all pods are Running or Completed
kubectl get pods -n pca
# Confirm node status
kubectl get nodes
# Confirm NTP
timedatectl status
The Admin Console should show all services in a Ready state with data flowing into Inventory and Sessions.
Pod Issues
Pod problems are the most common issues you'll encounter. This section covers the main pod states that indicate problems and how to resolve them.
Pods Stuck in Pending
Diagnosis:
kubectl describe pod <pod-name> -n pca
Common Causes and Fixes:
The following table maps pending pod causes to their resolutions. Check the Events section of kubectl describe pod to identify which cause applies:
| Cause | Fix |
|---|---|
| Insufficient CPU/Memory | Scale up nodes or reduce resource requests |
| PVC not bound | Check StorageClass availability |
| Node selector mismatch | Verify node labels |
| Taints preventing scheduling | Add tolerations or remove taints |
Check PVC status:
kubectl get pvc -n pca
kubectl describe pvc <pvc-name> -n pca
Pods in CrashLoopBackOff
This means the pod is starting, crashing, and restarting repeatedly. The key is to check the logs from the previous container instance.
Diagnosis:
# Get current logs
kubectl logs <pod-name> -n pca
# Get logs from previous crash
kubectl logs <pod-name> -n pca --previous
# Check events
kubectl describe pod <pod-name> -n pca
Common Causes:
- Missing configuration or secrets
- Database connection failures
- Out of memory (OOMKilled)
- Application startup errors
Pods in ImagePullBackOff
Symptom: Pod can't pull container image.
Diagnosis:
kubectl describe pod <pod-name> -n pca | grep -A 5 "Events"
Common Causes:
- Invalid image pull secret
- Network connectivity to registry
- Image doesn't exist
Database Issues
Database problems affect multiple services. If you're seeing widespread failures, check the databases first.
PostgreSQL Connection Failures
Symptoms: Multiple services failing, authentication errors, configuration not loading.
Diagnosis:
# Check PostgreSQL pod
kubectl get pods -n pca -l app=postgres
kubectl logs -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}')
# Test connectivity
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -c "SELECT 1;"
Common Fixes:
- Check PVC is bound and has space
- Verify PostgreSQL pod has enough memory
- Check for connection limit exhaustion
Druid Issues
Symptoms: No data in dashboards, slow queries, historical data missing.
Diagnosis:
# Check all Druid components
kubectl get pods -n pca | grep druid
# Check coordinator status
kubectl port-forward -n pca svc/druid-coordinator 8081:8081 &
curl http://localhost:8081/status
# Check supervisor status
curl http://localhost:8081/druid/indexer/v1/supervisor
Common Fixes:
- Restart failed supervisors
- Check MinIO connectivity for deep storage
- Verify Kafka topics exist and have data
Kafka Issues
Kafka is the backbone of the data pipeline. If Kafka is unhealthy, no new data will flow through the system.
High Consumer Lag
Symptoms: Data delays, dashboards showing stale data.
Diagnosis:
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \
kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups
Common Fixes:
- Check consumer pod health (Ignite, Druid indexers)
- Verify consumer pods have enough resources
- Scale up consumer replicas if needed
Kafka Broker Not Ready
Diagnosis:
# Check Kafka pods
kubectl get pods -n pca -l app=kafka
# Check Kafka logs
kubectl logs -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}')
Common Causes:
- Zookeeper connectivity issues
- Disk space exhaustion
- Memory pressure
Certificate Issues
Certificate problems cause secure connections to fail. They can affect both internal service communication and external access.
Certificate Expired or Invalid
Symptoms: TLS errors, services can't communicate, browser certificate warnings.
Diagnosis:
# Check certificate status
kubectl get certificates -n pca
# Check specific certificate
kubectl describe certificate <cert-name> -n pca
# Check expiration
kubectl get secret -n pca nginx-tls-secret-cm -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate
Common Fixes:
- Check cert-manager is running
- Verify ClusterIssuer is configured correctly
- Delete certificate to trigger renewal
Network Issues
Network problems can manifest in many ways. Use these checks when you suspect connectivity issues.
Services Can't Communicate
Diagnosis:
# Test DNS resolution
kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- nslookup gather
# Test service connectivity
kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- curl -v http://gather/health
External Access Not Working
Diagnosis:
# Check ingress
kubectl get ingress -n pca
# Check nginx pod
kubectl get pods -n pca -l app=nginx
kubectl logs -n pca $(kubectl get pods -n pca -l app=nginx -o jsonpath='{.items[0].metadata.name}')
Authentication Issues
Authentication problems prevent users from logging in. Zitadel is the identity provider for PCA.
Users Can't Log In
Diagnosis:
# Check Zitadel pod
kubectl get pods -n pca -l app=zitadel
kubectl logs -n pca $(kubectl get pods -n pca -l app=zitadel -o jsonpath='{.items[0].metadata.name}')
Common Fixes:
- Verify Zitadel database connectivity
- Check SMTP configuration for password reset
- Verify domain configuration matches DNS
Default Admin Password Not Working
For Crosswork Assurance 25.07 CA and later:
- Password is displayed in Admin Console
- Check Admin Console dashboard for credentials
Password Reset:
# Access Zitadel UI
# DNS mode: https://auth.{domain-name}
# No-DNS mode: https://{external-ip}:3443
Browser Displays Loading Screen
Symptoms: The Crosswork Assurance loading screen progress bar stops partway through. The browser console shows a failed feature-flags request or a "Not authorized" error. This can occur on Chrome, Edge, and other browsers.
Cause: The browser has not accepted the Zitadel authentication service certificate, causing the session to fail silently on the login redirect.
Resolution:
- Open a new browser tab and navigate to the Zitadel authentication page:
- DNS mode:
https://auth.<domain-name> - No-DNS mode:
https://<external-ip>:3443
- DNS mode:
- If prompted with a certificate warning, accept it to proceed. You do not need to log in to Zitadel.
- Return to the Crosswork Assurance URL and reload the page.
If the issue persists, clear your browser cookies or cache for the current browsing session and try again.
Performance Issues
Performance problems are often related to resource constraints or cache warming.
Slow Queries
Symptoms: UI dashboards load slowly, API timeouts.
Diagnosis:
# Check Druid historical cache
kubectl port-forward -n pca svc/health-clinic 8080:80 &
curl http://localhost:8080/metrics | grep druid_historical_cache
# Check Druid query metrics
curl http://localhost:8080/metrics | grep druid_query
Common Fixes:
- Wait for historical cache warm-up after restart
- Check Druid resource allocation
- Verify data retention policies
High Memory Usage
Diagnosis:
# Check pod resource usage
kubectl top pods -n pca --sort-by=memory
# Check node resources
kubectl top nodes
Log Collection
When you need to escalate or investigate further, collect these logs:
Collecting Service Logs
# Single service logs
kubectl logs -n pca <pod-name> > <service>-logs.txt
# All pods in namespace
for pod in $(kubectl get pods -n pca -o jsonpath='{.items[*].metadata.name}'); do
kubectl logs -n pca $pod > logs/$pod.txt 2>&1
done
Creating a Support Bundle
The Admin Console can generate a comprehensive support bundle that includes logs, configuration, and diagnostic information:
- Access the Admin Console.
- Navigate to the Troubleshoot section.
- Click Analyze to generate a comprehensive support bundle.
- Download the bundle and share it with Cisco support.
Collecting Events
kubectl get events -n pca --sort-by='.lastTimestamp' > pca-events.txt
Escalation Checklist
Before escalating to Cisco support, ensure you have collected the following information. This will help support resolve your issue faster:
- Support bundle from Admin Console
helmStderrlogs from failed deployment- Output of
kubectl get pods -n pca -o wide - Output of
kubectl get events -n pca - Relevant service logs
- Crosswork Assurance version and deployer version
- Deployment type (VM/K8s, single/multi-node)
- Network configuration (DNS/No-DNS, IPv4/IPv6)
Quick Reference Commands
Keep these commands handy for rapid troubleshooting:
# Overall health
kubectl get pods -n pca
kubectl get deployments -n pca
kubectl get statefulsets -n pca
# Logs
kubectl logs -n pca <pod-name>
kubectl logs -n pca <pod-name> --previous
# Describe resources
kubectl describe pod -n pca <pod-name>
kubectl describe pvc -n pca <pvc-name>
# Events
kubectl get events -n pca --sort-by='.lastTimestamp'
# Resource usage
kubectl top pods -n pca
kubectl top nodes
# Exec into pod
kubectl exec -it -n pca <pod-name> -- /bin/sh
© 2026 Cisco and/or its affiliates. All rights reserved.
For more information about trademarks, please visit: Cisco trademarks
For more information about legal terms, please visit: Cisco legal terms