Post-Installation Health Verification Guide

This article is intended to assist Customer Experience Engineers, Network Engineers, Field Engineers, with instructions to verify Crosswork Assurance platform health after an on-premise installation. It provides a standardized procedure to ensure all core components, databases, and data pipelines are functioning correctly following the initial deployment or upgrade of the Crosswork Assurance environment.

Use this guide immediately after installation, after upgrades, or whenever you suspect platform issues. A healthy platform means all pods are running, data is flowing through the pipeline, and users can access the system.

Quick Health Check Commands

Start here for a rapid assessment. These commands give you an immediate picture of platform state and should be your first step in any health verification.

1. Verify All Pods Are Running

kubectl get pods -n pca -o wide

Expected: All pods should show Running or Completed status. No pods should be in CrashLoopBackOff, Error, or Pending.

2. Check Core Services Health

# Check all deployments are ready
kubectl get deployments -n pca

# Check statefulsets (databases, Kafka, etc.)
kubectl get statefulsets -n pca

Expected: All deployments should show READY matching DESIRED (e.g., 1/1, 3/3).

3. Verify Ingress/External Access

# Check nginx ingress
kubectl get pods -n pca -l app=nginx

# Check service endpoints
kubectl get svc -n pca | grep -E "(nginx|LoadBalancer)"

Service-Level Health Endpoints

Each Crosswork Assurance service exposes health endpoints you can query directly. Use these when you need to diagnose issues with specific services or verify that a service is truly healthy beyond just "Running" status.

# Port-forward to a service and check health
kubectl port-forward -n pca svc/gather 8080:80 &
curl http://localhost:8080/health

# Check metrics endpoint
kubectl port-forward -n pca svc/gather 9191:9191 &
curl http://localhost:9191/metrics

Core Infrastructure Health

These are the foundational services that everything else depends on. If any of these are unhealthy, you'll see cascading failures across the platform. Check these first when troubleshooting widespread issues.

PostgreSQL Database

PostgreSQL stores configuration, user data, and application state. If it's down, authentication and most API operations will fail.

# Check PostgreSQL pod status
kubectl get pods -n pca -l app=postgres

# Connect and verify
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -c "SELECT 1;"

Expected: Returns 1 indicating the database is responsive.

Druid Analytics Database

Druid handles all time-series data storage and queries. If Druid is unhealthy, dashboards will show no data or queries will time out.

# Check Druid components
kubectl get pods -n pca -l app=druid-coordinator
kubectl get pods -n pca -l app=druid-broker
kubectl get pods -n pca -l app=druid-historical

# Check coordinator status
kubectl port-forward -n pca svc/druid-coordinator 8081:8081 &
curl http://localhost:8081/status

Expected: All Druid pods running, coordinator returns status JSON.

Redis/Valkey Cache

Redis provides caching and session storage. Issues here typically manifest as slow performance or session problems.

# Check Redis pod
kubectl get pods -n pca -l app=redis

# Test connectivity
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=redis -o jsonpath='{.items[0].metadata.name}') -- redis-cli ping

Expected: Returns PONG.

Kafka Cluster

Kafka is the backbone of the data pipeline. All telemetry flows through Kafka topics. If Kafka is unhealthy, no new data will appear in the system.

# Check Kafka pods
kubectl get pods -n pca -l app=kafka

# List Kafka topics (verify data flow)
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \
  kafka-topics.sh --list --bootstrap-server localhost:9092

Expected Topics: spark_raw_in.*, spark_latent_in.*, normalized.*, and alert-events.

Health Clinic Service

Health Clinic is Crosswork Assurance's internal monitoring service that aggregates health metrics from all components. It's your single pane of glass for platform health and the source of data for the Grafana dashboards.

The following table shows the key metrics Health Clinic exposes. Use these to understand overall platform health at a glance:

Data Flow Verification

This is where you verify that data is actually moving through the system end-to-end. A platform can have all pods running but still have a broken data pipeline. Use these checks to confirm telemetry is flowing from sensors to the UI.

The expected latency from Sensor to UI is approximately 10 seconds.

# Expected Data Flow:
Sensor/Collector -> RoadRunner -> Fedex -> Kafka -> Ignite -> Druid -> Gather -> UI

Expected Data Flow Diagram

This diagram shows the complete data path through the platform. When troubleshooting missing data, trace backwards from the UI to find where the flow is broken:

Verify Kafka Consumer Lag

Consumer lag tells you if data is backing up in Kafka. High lag means consumers can't keep up with incoming data.

kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \
  kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups

Expected: Consumer lag should be low (fewer than 1000 messages) under normal operation.

Verify Druid Ingestion

Druid supervisors manage the ingestion of data from Kafka into Druid segments. If supervisors aren't running, historical data won't be available.

# Check supervisor status
kubectl port-forward -n pca svc/druid-coordinator 8081:8081 &
curl http://localhost:8081/druid/indexer/v1/supervisor

# Check running tasks
curl http://localhost:8081/druid/indexer/v1/runningTasks

Expected: Supervisors in RUNNING state, active ingestion tasks present.

Certificate Verification

TLS certificates secure communication between components and with external clients. Expired or invalid certificates cause connection failures that can be difficult to diagnose.

# Check certificate status
kubectl get certificates -n pca

# Verify certificate expiration
kubectl get secret -n pca nginx-tls-secret-cm -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

# Check cert-manager
kubectl get pods -n cert-manager

Expected: All certificates should show Ready=True and have valid expiration dates.

Common Post-Installation Issues

These are the issues we see most frequently after new installations or upgrades. Use this table to quickly identify likely causes based on symptoms:

Health Check Script

Use this script for a quick overall health assessment. It checks the most critical components and highlights any issues:

#!/bin/bash
NAMESPACE="${1:-pca}"
echo "=== Provider Connectivity Assurance Health Check ==="
echo ""
echo "1. Pod Status:"
kubectl get pods -n $NAMESPACE --no-headers | awk '{print $3}' | sort | uniq -c
echo ""
echo "2. Deployments Not Ready:"
kubectl get deployments -n $NAMESPACE | grep -v "1/1\|2/2\|3/3" | grep -v NAME
echo ""
echo "3. StatefulSets Not Ready:"
kubectl get statefulsets -n $NAMESPACE | grep -v "1/1\|2/2\|3/3" | grep -v NAME
echo ""
echo "4. Recent Events (last 10):"
kubectl get events -n $NAMESPACE --sort-by='.lastTimestamp' | tail -10
echo ""
echo "5. PVC Status:"
kubectl get pvc -n $NAMESPACE | grep -v Bound
echo ""
echo "=== Health Check Complete ==="