On-Prem Troubleshooting

This article is intended to assist Customer Experience Engineers, Network Engineers, Field Engineers to diagnose and resolve common Crosswork Assurance on-premise installation and operational issues. It's organized by symptom so you can quickly find the right troubleshooting steps based on what you're seeing.

Troubleshooting Methodology

Before diving into specific issues, follow this systematic approach. It will save you time and help you avoid chasing symptoms instead of root causes:

1. Identify symptoms: Determine exactly what the user is experiencing.

2. Check platform health: Run standard health verification commands.

3. Isolate the layer: Determine if the issue is Infrastructure, Service, or Application related.

4. Collect logs: Gather relevant diagnostic information from the affected components.

5. Apply fix: Follow runbook steps or escalate to the next tier.

Pre-Flight Check Failures

These issues occur during installation before Crosswork Assurance is fully deployed. They're usually related to missing prerequisites or environment configuration.

Error: socat not found

Symptom: Installation fails with unable to do port forwarding: socat not found.

Cause: The socat package is missing from the Kubernetes nodes.

Resolution:

# RHEL/CentOS
sudo yum install -y socat

# Ubuntu/Debian
sudo apt-get install -y socat

# Verify installation
which socat

Error: IPv6/Dual-Stack Mismatch

Symptom: Pre-flight checks fail indicating a dual-stack version is required.

Cause: An IPv6 address was detected but a non-dual-stack deployer version was selected.

Resolution: Download the correct dual-stack version (e.g., 25.7.250-105-dualstack) and re-run the installation.

Error: Insufficient Resources

Symptom: Pre-flight checks fail with resource warnings.

Cause: Node does not meet minimum requirements (32 vCPU, 64GB RAM minimum per node).

Resolution:

1. Verify node resources: kubectl describe node <node-name>

2. Scale up VM resources or add additional nodes

3. Re-run pre-flight checks

Error: RBAC Permission Denied

Symptom: Installation fails with permission errors.

Cause: User lacks cluster-admin permissions.

Resolution:

# Grant cluster-admin to current user
kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole=cluster-admin \
  --user=$(gcloud config get-value account)

Pod Issues

Pod problems are the most common issues you'll encounter. This section covers the main pod states that indicate problems and how to resolve them.

Pods Stuck in Pending

Diagnosis:

kubectl describe pod <pod-name> -n pca

Common Causes and Fixes:

The following table maps pending pod causes to their resolutions. Check the Events section of kubectl describe pod to identify which cause applies:

Check PVC status:

kubectl get pvc -n pca
kubectl describe pvc <pvc-name> -n pca

Pods in CrashLoopBackOff

This means the pod is starting, crashing, and restarting repeatedly. The key is to check the logs from the previous container instance.

Diagnosis:

# Get current logs
kubectl logs <pod-name> -n pca

# Get logs from previous crash
kubectl logs <pod-name> -n pca --previous

# Check events
kubectl describe pod <pod-name> -n pca

Common Causes:

• Missing configuration or secrets

• Database connection failures

• Out of memory (OOMKilled)

• Application startup errors

Pods in ImagePullBackOff

Symptom: Pod can't pull container image.

Diagnosis:

kubectl describe pod <pod-name> -n pca | grep -A 5 "Events"

Common Causes:

• Invalid image pull secret

• Network connectivity to registry

• Image doesn't exist

Database Issues

Database problems affect multiple services. If you're seeing widespread failures, check the databases first.

PostgreSQL Connection Failures

Symptoms: Multiple services failing, authentication errors, configuration not loading.

Diagnosis:

# Check PostgreSQL pod
kubectl get pods -n pca -l app=postgres
kubectl logs -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}')

# Test connectivity
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -c "SELECT 1;"

Common Fixes:

1. Check PVC is bound and has space

2. Verify PostgreSQL pod has enough memory

3. Check for connection limit exhaustion

Druid Issues

Symptoms: No data in dashboards, slow queries, historical data missing.

Diagnosis:

# Check all Druid components
kubectl get pods -n pca | grep druid

# Check coordinator status
kubectl port-forward -n pca svc/druid-coordinator 8081:8081 &
curl http://localhost:8081/status

# Check supervisor status
curl http://localhost:8081/druid/indexer/v1/supervisor

Common Fixes:

1. Restart failed supervisors

2. Check MinIO connectivity for deep storage

3. Verify Kafka topics exist and have data

Kafka Issues

Kafka is the backbone of the data pipeline. If Kafka is unhealthy, no new data will flow through the system.

High Consumer Lag

Symptoms: Data delays, dashboards showing stale data.

Diagnosis:

kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \
  kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups

Common Fixes:

1. Check consumer pod health (Ignite, Druid indexers)

2. Verify consumer pods have enough resources

3. Scale up consumer replicas if needed

Kafka Broker Not Ready

Diagnosis:

# Check Kafka pods
kubectl get pods -n pca -l app=kafka

# Check Kafka logs
kubectl logs -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}')

Common Causes:

• Zookeeper connectivity issues

• Disk space exhaustion

• Memory pressure

Certificate Issues

Certificate problems cause secure connections to fail. They can affect both internal service communication and external access.

Certificate Expired or Invalid

Symptoms: TLS errors, services can't communicate, browser certificate warnings.

Diagnosis:

# Check certificate status
kubectl get certificates -n pca

# Check specific certificate
kubectl describe certificate <cert-name> -n pca

# Check expiration
kubectl get secret -n pca nginx-tls-secret-cm -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

Common Fixes:

1. Check cert-manager is running

2. Verify ClusterIssuer is configured correctly

3. Delete certificate to trigger renewal

Network Issues

Network problems can manifest in many ways. Use these checks when you suspect connectivity issues.

Services Can't Communicate

Diagnosis:

# Test DNS resolution
kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- nslookup gather

# Test service connectivity
kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- curl -v http://gather/health

External Access Not Working

Diagnosis:

# Check ingress
kubectl get ingress -n pca

# Check nginx pod
kubectl get pods -n pca -l app=nginx
kubectl logs -n pca $(kubectl get pods -n pca -l app=nginx -o jsonpath='{.items[0].metadata.name}')

Authentication Issues

Authentication problems prevent users from logging in. Zitadel is the identity provider for PCA.

Users Can't Log In

Diagnosis:

# Check Zitadel pod
kubectl get pods -n pca -l app=zitadel
kubectl logs -n pca $(kubectl get pods -n pca -l app=zitadel -o jsonpath='{.items[0].metadata.name}')

Common Fixes:

1. Verify Zitadel database connectivity

2. Check SMTP configuration for password reset

3. Verify domain configuration matches DNS

Default Admin Password Not Working

For Crosswork Assurance 25.07 CA and later:

• Password is displayed in Admin Console

• Check Admin Console dashboard for credentials

Password Reset:

# Access Zitadel UI
# DNS mode: https://auth.{domain-name}
# No-DNS mode: https://{external-ip}:3443

Performance Issues

Performance problems are often related to resource constraints or cache warming.

Slow Queries

Symptoms: UI dashboards load slowly, API timeouts.

Diagnosis:

# Check Druid historical cache
kubectl port-forward -n pca svc/health-clinic 8080:80 &
curl http://localhost:8080/metrics | grep druid_historical_cache

# Check Druid query metrics
curl http://localhost:8080/metrics | grep druid_query

Common Fixes:

1. Wait for historical cache warm-up after restart

2. Check Druid resource allocation

3. Verify data retention policies

High Memory Usage

Diagnosis:

# Check pod resource usage
kubectl top pods -n pca --sort-by=memory

# Check node resources
kubectl top nodes

Log Collection

When you need to escalate or investigate further, collect these logs:

Collecting Service Logs

# Single service logs
kubectl logs -n pca <pod-name> > <service>-logs.txt

# All pods in namespace
for pod in $(kubectl get pods -n pca -o jsonpath='{.items[*].metadata.name}'); do
  kubectl logs -n pca $pod > logs/$pod.txt 2>&1
done

Creating a Support Bundle

The Admin Console can generate a comprehensive support bundle that includes logs, configuration, and diagnostic information:

1. Access the Admin Console.

2. Navigate to the Troubleshoot section.

3. Click Analyze to generate a comprehensive support bundle.

4. Download the bundle and share it with Cisco support.

Collecting Events

kubectl get events -n pca --sort-by='.lastTimestamp' > pca-events.txt

Escalation Checklist

Before escalating to Cisco support, ensure you have collected the following information. This will help support resolve your issue faster:

• Support bundle from Admin Console

• helmStderr logs from failed deployment

• Output of kubectl get pods -n pca -o wide

• Output of kubectl get events -n pca

• Relevant service logs

• Crosswork Assurance version and deployer version

• Deployment type (VM/K8s, single/multi-node)

• Network configuration (DNS/No-DNS, IPv4/IPv6)

Quick Reference Commands

Keep these commands handy for rapid troubleshooting:

# Overall health
kubectl get pods -n pca
kubectl get deployments -n pca
kubectl get statefulsets -n pca

# Logs
kubectl logs -n pca <pod-name>
kubectl logs -n pca <pod-name> --previous

# Describe resources
kubectl describe pod -n pca <pod-name>
kubectl describe pvc -n pca <pvc-name>

# Events
kubectl get events -n pca --sort-by='.lastTimestamp'

# Resource usage
kubectl top pods -n pca
kubectl top nodes

# Exec into pod
kubectl exec -it -n pca <pod-name> -- /bin/sh