CLI Reference Guide

This article is intended to assist Customer Experience Engineers, Network Engineers, Field Engineers, with essential command-line tools and commands for administering and troubleshooting the Crosswork Assurance platform. This reference gives you the essential command-line tools and commands for administering and troubleshooting Crosswork Assurance. Keep it handy when you're working on the platform—most troubleshooting starts with these commands.

Overview

You'll primarily use three CLI tools when working with Crosswork Assurance. Here's what each one does:

Crosswork Assurance CLI

The pca CLI is available within the cluster for sensor and configuration operations. Use it when you need to manage sensors or check tenant configurations.

Accessing the CLI

# Exec into a service pod that has the CLI
kubectl exec -it -n pca deploy/sensor-orchestrator -- /bin/sh

# Or use a debug pod
kubectl run pca-debug --rm -it --image=pca-tools:latest -n pca -- /bin/sh

Common CLI Commands

# List registered sensors
pca sensor list

# Get sensor details
pca sensor get <sensor-id>

# Check sensor configuration
pca config show <tenant-id>

# Validate configuration
pca config validate <config-file>

kubectl Commands

kubectl is your primary tool for interacting with the Kubernetes cluster. These are the commands you'll use most frequently.

Pod Management

Pods are the basic unit of deployment in Kubernetes. These commands help you check pod status and manage pod lifecycle:

# List all pods
kubectl get pods -n pca

# List pods with more details
kubectl get pods -n pca -o wide

# Watch pod status changes
kubectl get pods -n pca -w

# Get pod details
kubectl describe pod <pod-name> -n pca

# Delete a pod (will be recreated by deployment)
kubectl delete pod <pod-name> -n pca

Logs

Logs are essential for troubleshooting. These commands help you access logs in different ways:

# Get current logs
kubectl logs <pod-name> -n pca

# Follow logs in real-time
kubectl logs -f <pod-name> -n pca

# Get logs from previous container instance
kubectl logs <pod-name> -n pca --previous

# Get logs from specific container in multi-container pod
kubectl logs <pod-name> -n pca -c <container-name>

# Get last 100 lines
kubectl logs <pod-name> -n pca --tail=100

# Get logs from last hour
kubectl logs <pod-name> -n pca --since=1h

Exec into Pods

Sometimes you need to get inside a pod to run commands or check files:

# Interactive shell
kubectl exec -it <pod-name> -n pca -- /bin/sh

# Run single command
kubectl exec <pod-name> -n pca -- ls -la /app

# Exec into specific container
kubectl exec -it <pod-name> -n pca -c <container-name> -- /bin/sh

Port Forwarding

Port forwarding lets you access services running in the cluster from your local machine:

# Forward local port to service
kubectl port-forward -n pca svc/gather 8080:80

# Forward to specific pod
kubectl port-forward -n pca <pod-name> 8080:80

# Run in background
kubectl port-forward -n pca svc/gather 8080:80 &

Resource Status

These commands help you check the status of various Kubernetes resources:

# Deployments
kubectl get deployments -n pca
kubectl describe deployment <name> -n pca

# StatefulSets
kubectl get statefulsets -n pca
kubectl describe statefulset <name> -n pca

# Services
kubectl get svc -n pca
kubectl describe svc <name> -n pca

# ConfigMaps
kubectl get configmaps -n pca
kubectl describe configmap <name> -n pca

# Secrets (names only)
kubectl get secrets -n pca

# PVCs
kubectl get pvc -n pca
kubectl describe pvc <name> -n pca

# Events
kubectl get events -n pca --sort-by='.lastTimestamp'

Scaling

Use these commands to scale deployments or restart them:

# Scale deployment
kubectl scale deployment <name> -n pca --replicas=3

# Restart deployment (rolling restart)
kubectl rollout restart deployment <name> -n pca

# Check rollout status
kubectl rollout status deployment <name> -n pca

KOTS Admin Console CLI

The KOTS CLI manages the Admin Console and application lifecycle. Use it for installation management, upgrades, and support bundle generation.

Installation

# Install KOTS CLI
curl https://kots.io/install | bash

# Verify installation
kubectl kots version

Common KOTS Commands

# Access Admin Console (opens browser)
kubectl kots admin-console -n pca

# Get Admin Console password
kubectl kots admin-console --password -n pca

# Download support bundle
kubectl kots support-bundle -n pca

# Check for updates
kubectl kots upstream upgrade -n pca

# View application status
kubectl kots get apps -n pca

Database CLI Commands

These commands help you interact with PCA's databases for troubleshooting and verification.

PostgreSQL

PostgreSQL stores configuration, users, and application state:

# Connect to PostgreSQL
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres

# Run a quick connectivity test
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -c "SELECT 1;"

# List databases
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=postgres -o jsonpath='{.items[0].metadata.name}') -- psql -U postgres -c "\l"

Redis/Valkey

Redis provides caching and session storage:

# Connect to Redis
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=redis -o jsonpath='{.items[0].metadata.name}') -- redis-cli

# Test connectivity
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=redis -o jsonpath='{.items[0].metadata.name}') -- redis-cli ping

# Check memory usage
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=redis -o jsonpath='{.items[0].metadata.name}') -- redis-cli info memory

Kafka

Kafka is the message backbone for the data pipeline:

# List topics
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \
  kafka-topics.sh --list --bootstrap-server localhost:9092

# Describe a topic
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \
  kafka-topics.sh --describe --topic <topic-name> --bootstrap-server localhost:9092

# Check consumer lag
kubectl exec -it -n pca $(kubectl get pods -n pca -l app=kafka -o jsonpath='{.items[0].metadata.name}') -- \
  kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups

Druid

Druid handles time-series data storage and analytics queries:

# Port-forward to coordinator
kubectl port-forward -n pca svc/druid-coordinator 8081:8081 &

# Check cluster status
curl http://localhost:8081/status

# List datasources
curl http://localhost:8081/druid/coordinator/v1/datasources

# Check supervisor status
curl http://localhost:8081/druid/indexer/v1/supervisor

# Check running tasks
curl http://localhost:8081/druid/indexer/v1/runningTasks

# Check pending tasks
curl http://localhost:8081/druid/indexer/v1/pendingTasks

Certificate Management

These commands help you check and troubleshoot TLS certificates:

# List certificates
kubectl get certificates -n pca

# Describe certificate
kubectl describe certificate <cert-name> -n pca

# Check certificate expiration
kubectl get secret -n pca <secret-name> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate

# View certificate details
kubectl get secret -n pca <secret-name> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -text

# Check cert-manager logs
kubectl logs -n cert-manager $(kubectl get pods -n cert-manager -l app=cert-manager -o jsonpath='{.items[0].metadata.name}')

Health Check Commands

Use these commands for quick health assessments:

# Quick cluster health
kubectl get pods -n pca | grep -v Running | grep -v Completed

# Check all deployments ready
kubectl get deployments -n pca -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.readyReplicas}/{.status.replicas}{"\n"}{end}'

# Check resource usage
kubectl top pods -n pca
kubectl top nodes

# Check events for errors
kubectl get events -n pca --field-selector type=Warning

Useful Aliases

Add these to your ~/.bashrc or ~/.zshrc to speed up your workflow:

# Kubernetes shortcuts
alias k='kubectl'
alias kpca='kubectl -n pca'
alias kgp='kubectl get pods -n pca'
alias kgpw='kubectl get pods -n pca -w'
alias kgd='kubectl get deployments -n pca'
alias kgs='kubectl get svc -n pca'
alias kge='kubectl get events -n pca --sort-by=.lastTimestamp'

# Quick log access
alias klogs='kubectl logs -n pca'
alias klogsf='kubectl logs -f -n pca'

# Exec shortcuts
alias kexec='kubectl exec -it -n pca'

# Port-forward shortcuts
alias kpf='kubectl port-forward -n pca'

Troubleshooting Commands

These commands help you diagnose common issues:

# Find pods not running
kubectl get pods -n pca --field-selector=status.phase!=Running,status.phase!=Succeeded

# Get pod restart counts
kubectl get pods -n pca -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.restartCount}{" "}{end}{"\n"}{end}'

# Check for resource constraints
kubectl describe nodes | grep -A 5 "Allocated resources"

# Get all container images
kubectl get pods -n pca -o jsonpath='{range .items[*]}{range .spec.containers[*]}{.image}{"\n"}{end}{end}' | sort | uniq

# Check PVC usage
kubectl exec -it -n pca <pod-with-pvc> -- df -h

# Network debugging
kubectl run netdebug --rm -it --image=nicolaka/netshoot -n pca -- /bin/bash

Command Quick Reference

Keep this table handy for the most common tasks: