--- title: "Post-Installation Health Verification Guide" slug: "post-installation-health-verification-guide" updated: 2026-10-22T21:23:59Z published: 2026-10-22T21:23:59Z canonical: "docs.crossworkassurance.cisco.com/post-installation-health-verification-guide" --- > ## Documentation Index > Fetch the complete documentation index at: https://docs.crossworkassurance.cisco.com/llms.txt > Use this file to discover all available pages before exploring further. # Post-Installation Health Verification Guide This article is intended to assist Customer Experience Engineers, Network Engineers, Field Engineers, with instructions to verify Crosswork Assurance platform health after an on-premise installation. It provides a standardized procedure to ensure all core components, databases, and data pipelines are functioning correctly following the initial deployment or upgrade of the Crosswork Assurance environment. Use this guide immediately after installation, after upgrades, or whenever you suspect platform issues. A healthy platform means all pods are running, data is flowing through the pipeline, and users can access the system. ### Quick Health Check Commands Start here for a rapid assessment. These commands give you an immediate picture of platform state and should be your first step in any health verification. #### 1. Verify All Pods Are Running ``` kubectl get pods -n pca -o wide ``` **Expected:** All pods should show `Running` or `Completed` status. No pods should be in `CrashLoopBackOff`, `Error`, or `Pending`. #### 2. Check Core Services Health ``` # Check all deployments are ready kubectl get deployments -n pca # Check statefulsets (databases, Kafka, etc.) kubectl get statefulsets -n pca ``` **Expected:** All deployments should show `READY` matching `DESIRED` (e.g., 1/1, 3/3). #### 3. Verify Ingress/External Access ``` # Check nginx ingress kubectl get pods -n pca -l app=nginx # Check service endpoints kubectl get svc -n pca | grep -E "(nginx|LoadBalancer)" ``` ### Service-Level Health Endpoints Each Crosswork Assurance service exposes health endpoints you can query directly. Use these when you need to diagnose issues with specific services or verify that a service is truly healthy beyond just "Running" status. | Endpoint | Purpose | Expected Response | | --- | --- | --- | | `/health` | Liveness check | 200 OK | | `/ready` | Readiness check | 200 OK when ready to serve traffic | | `:9191/metrics` | Prometheus metrics | Prometheus-formatted metrics | ``` # Port-forward to a service and check health kubectl port-forward -n pca svc/gather 8080:80 & curl http://localhost:8080/health # Check metrics endpoint kubectl port-forward -n pca svc/gather 9191:9191 & curl http://localhost:9191/metrics ``` --- ### Core Infrastructure Health These are the foundational services that everything else depends on. If any of these are unhealthy, you'll see cascading failures across the platform. Check these first when troubleshooting widespread issues. #### PostgreSQL Database PostgreSQL stores configuration, user data, and application state. If it's down, authentication and most API operations will fail. ``` # Check PostgreSQL pod status kubectl get pods -n pca -l app=postgres # Connect and verify kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -c "SELECT 1;" ``` **Expected:** Returns `1` indicating the database is responsive. #### Druid Analytics Database Druid handles all time-series data storage and queries. If Druid is unhealthy, dashboards will show no data or queries will time out. ``` # Check Druid components kubectl get pods -n pca -l app=druid-coordinator kubectl get pods -n pca -l app=druid-broker kubectl get pods -n pca -l app=druid-historical # Check coordinator status kubectl port-forward -n pca svc/druid-coordinator 8081:8081 & curl http://localhost:8081/status ``` **Expected:** All Druid pods running, coordinator returns status JSON. #### Redis/Valkey Cache Redis provides caching and session storage. Issues here typically manifest as slow performance or session problems. ``` # Check Redis pod kubectl get pods -n pca -l app=redis # Test connectivity kubectl exec -it -n pca $(kubectl get pods -n pca -l app=redis -o jsonpath='{.items[0].metadata.name}') -- redis-cli ping ``` **Expected:** Returns `PONG`. #### Kafka Cluster Kafka is the backbone of the data pipeline. All telemetry flows through Kafka topics. If Kafka is unhealthy, no new data will appear in the system. ``` # Check Kafka pods kubectl get pods -n pca -l app=kafka # List Kafka topics (verify data flow) kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \ kafka-topics.sh --list --bootstrap-server localhost:9092 ``` **Expected Topics:** `spark_raw_in.*`, `spark_latent_in.*`, `normalized.*`, and `alert-events`. --- ### Health Clinic Service Health Clinic is Crosswork Assurance's internal monitoring service that aggregates health metrics from all components. It's your single pane of glass for platform health and the source of data for the Grafana dashboards. The following table shows the key metrics Health Clinic exposes. Use these to understand overall platform health at a glance: | Metric Category | What It Tells You | Where to Check | | --- | --- | --- | | Kafka Metrics | Consumer lag, throughput | Grafana → Kafka dashboard | | Druid Metrics | Query performance, ingestion | Grafana → Druid dashboard | | Infrastructure | Node resources, pod status | Grafana → Infrastructure dashboard | ### Data Flow Verification This is where you verify that data is actually moving through the system end-to-end. A platform can have all pods running but still have a broken data pipeline. Use these checks to confirm telemetry is flowing from sensors to the UI. The expected latency from Sensor to UI is approximately 10 seconds. ``` # Expected Data Flow: Sensor/Collector -> RoadRunner -> Fedex -> Kafka -> Ignite -> Druid -> Gather -> UI ``` #### Expected Data Flow Diagram This diagram shows the complete data path through the platform. When troubleshooting missing data, trace backwards from the UI to find where the flow is broken: ![](https://cdn.document360.io/01d1de5e-a375-4fbc-aa82-666ac04c2113/Images/Documentation/image-URMNMJNS.png?sv=2026-02-06&spr=https&st=2026-11-19T19%3A50%3A44Z&se=2026-11-19T20%3A50%3A44Z&sr=b&sp=r&sig=QOMjYf8IM9a5UdgJ6EnCv9z2Z4Uq2GAOmpOw6d0hxX8%3D) #### Verify Kafka Consumer Lag Consumer lag tells you if data is backing up in Kafka. High lag means consumers can't keep up with incoming data. ``` kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups ``` **Expected:** Consumer lag should be low (fewer than 1000 messages) under normal operation. #### Verify Druid Ingestion Druid supervisors manage the ingestion of data from Kafka into Druid segments. If supervisors aren't running, historical data won't be available. ``` # Check supervisor status kubectl port-forward -n pca svc/druid-coordinator 8081:8081 & curl http://localhost:8081/druid/indexer/v1/supervisor # Check running tasks curl http://localhost:8081/druid/indexer/v1/runningTasks ``` **Expected:** Supervisors in `RUNNING` state, active ingestion tasks present. ### Certificate Verification TLS certificates secure communication between components and with external clients. Expired or invalid certificates cause connection failures that can be difficult to diagnose. ``` # Check certificate status kubectl get certificates -n pca # Verify certificate expiration kubectl get secret -n pca nginx-tls-secret-cm -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate # Check cert-manager kubectl get pods -n cert-manager ``` **Expected:** All certificates should show `Ready=True` and have valid expiration dates. ### Common Post-Installation Issues These are the issues we see most frequently after new installations or upgrades. Use this table to quickly identify likely causes based on symptoms: | Symptom | Likely Cause | Verification Command | | --- | --- | --- | | Pods stuck in Pending | Insufficient resources or PVC issues | `kubectl describe pod [name] -n pca` | | No data in UI | Kafka/Druid ingestion issue | Check Kafka consumer lag | | Pods in CrashLoopBackOff | Configuration or dependency error | `kubectl logs [pod] -n pca --previous` | | Certificate errors | cert-manager or issuer issue | `kubectl get certificates -n pca` | | Slow queries | Druid cache not warmed | Wait 10-15 min after restart | ### Health Check Script Use this script for a quick overall health assessment. It checks the most critical components and highlights any issues: ``` #!/bin/bash NAMESPACE="${1:-pca}" echo "=== Provider Connectivity Assurance Health Check ===" echo "" echo "1. Pod Status:" kubectl get pods -n $NAMESPACE --no-headers | awk '{print $3}' | sort | uniq -c echo "" echo "2. Deployments Not Ready:" kubectl get deployments -n $NAMESPACE | grep -v "1/1\|2/2\|3/3" | grep -v NAME echo "" echo "3. StatefulSets Not Ready:" kubectl get statefulsets -n $NAMESPACE | grep -v "1/1\|2/2\|3/3" | grep -v NAME echo "" echo "4. Recent Events (last 10):" kubectl get events -n $NAMESPACE --sort-by='.lastTimestamp' | tail -10 echo "" echo "5. PVC Status:" kubectl get pvc -n $NAMESPACE | grep -v Bound echo "" echo "=== Health Check Complete ===" ``` --- **© 2026 Cisco and/or its affiliates. All rights reserved.** For more information about trademarks, please visit: [Cisco trademarks](https://www.cisco.com/c/en/us/about/legal/trademarks.html)[](https://www.cisco.com/c/en/us/about/legal/trademarks.html) For more information about legal terms, please visit: [Cisco legal terms](https://www.cisco.com/c/en/us/about/legal/terms-conditions.html)