Documentation Index

Fetch the complete documentation index at: https://docs.crossworkassurance.cisco.com/llms.txt

Use this file to discover all available pages before exploring further.

New: Try our AI‑powered Search (Ctrl + K) — Read more

Understand the Framework

Prev Next

This article walks through the model we've built into the troubleshooting documentation. The goal is to give you a consistent framework for understanding the solution—not just a list of commands, but a way of thinking about the system.

Setting the stage: Before diving into specifics, here's a quick snapshot.

"Kubernetes-based platform with 60+ microservices that collects telemetry from sensors, processes it through a streaming pipeline, and serves insights via web UI and APIs."

The healthy state in one sentence:

  • All pods running ✓

  • Data flows sensor-to-UI in ~10 seconds ✓

  • Users can log in and see current metrics ✓

Three Questions Framework

The diagnostic starting point: We propose three questions to help you quickly isolate the problem domain.

Question

What It Reveals

Quick Check

Can users log in?

Authentication health

Try the UI login

Is data appearing?

Data pipeline health

Check Kafka consumer lag

Are pods running?

Infrastructure health

kubectl get pods -n pca

The logic:

  • If login fails → Focus on Presentation Plane (Zitadel, PostgreSQL)

  • If data is missing → Focus on Data Plane (Kafka, Ignite, Druid)

  • If pods are down → Focus on Infrastructure (resources, PVCs, configs)

Three Planes Architecture

Organizing 60+ services: We group services into three logical planes to make the system comprehensible.

Presentation Plane (User-Facing)

This is what you and your users interact with directly. When someone reports they "can't access the system" or "the UI is broken," start here. These services handle authentication, serve the web interface, and expose APIs.

Service

What It Does

Symptom When Broken

Gather

REST API gateway

API errors, timeouts

nginx

Serves UI

Can't access web interface

Zitadel

Authentication

Login failures

Grafana

Dashboards

Metrics dashboards unavailable

Data Plane (Telemetry Processing)

This is where the real work happens. Sensor data flows through these services to be ingested, processed, and stored. When users report "missing data," "stale metrics," or "dashboards not updating," you'll be investigating this plane.

Service

What It Does

Symptom When Broken

RoadRunner

Ingests sensor data

Sensors not connecting

Fedex

WebSocket broker

No real-time updates

Ignite

Stream processing

Data delays

Kafka

Message queue

Everything downstream fails

Druid

Time-series storage

Historical queries fail

Control Plane (Configuration)

These services manage how the platform is configured and how sensors are orchestrated. Issues here are often more subtle—sensors failing to register, configuration changes not taking effect, or tenant-specific problems. Check this plane when the system is "running" but not behaving as expected.

Service

What It Does

Symptom When Broken

Sensor-Orchestrator

Manages sensors

Sensors not registering

Config-Service

Configuration mgmt

Config not applying

Tenant-Manager

Multi-tenancy

Tenant issues

Data Flow Visualization

Tracing the happy path: We show how data moves through the system so you can trace backwards when something breaks.

Sensor → RoadRunner → Fedex → Kafka → Ignite → Druid → Gather → UI

Key insight: When data is missing, trace backwards:

  1. Is Druid ingesting? (Check supervisor status)

  2. Is Kafka flowing? (Check consumer lag)

  3. Is RoadRunner receiving? (Check pod logs)

  4. Is the sensor connected? (Check sensor status)

Expected latency: ~10 seconds end-to-end

Common Scenarios Mapping

Quick lookup: We map common symptoms to likely causes and first checks.

Scenario

Likely Plane

First Check

"Users can't log in"

Presentation

Zitadel pods, PostgreSQL

"No data in dashboards"

Data

Kafka lag, Druid supervisors

"Sensors not connecting"

Control

Certificates, RoadRunner logs

"Everything is slow"

Data

Druid cache, resource usage

"Pods keep restarting"

Infrastructure

Pod logs, resource limits

Reference: Full Documentation Set

The following guides provide detailed procedures for each area covered in this overview. Use them as your go-to references when you need step-by-step instructions or deeper technical detail.

Document

Purpose

Post-Installation Health Verification

Systematic health checks

Architecture Overview

Component relationships

On-Prem Troubleshooting Runbook

Diagnostic procedures

CLI Reference Guide

Command reference

Port Matrix & Firewall Rules

Network configuration

© 2026 Cisco and/or its affiliates. All rights reserved.

For more information about trademarks, please visit:
Cisco trademarks 
For more information about legal terms, please visit:
Cisco legal terms