Telemetry Ingestion Staging Feature

This article explains how to make use of the Telemetry Ingestion Staging feature in Crosswork Assurance.

Topics covered:

• How to enable the feature (Cloud and On-prem)
• Where to see your incoming Telemetry and Assurance Sensors data
• Getting Started: Understanding the Landing Page
• Setting up and customizing new object types and metrics in Crosswork Assurance
• Workflow Timesaver: Importing ingestion specifications across instances of Crosswork Assurance

Enabling the feature

Cloud deployments:

To turn on the feature, contact TAC via Intercom and request to have the Ingestion Staging feature enabled on the tenant.

On-prem deployments (Controlled Availability loads only, not in 25.07 GA):

To turn the ingestion staging feature on, log in to the Analytics UI using your deployment administrator user and navigate to the Settings > General > Feature Flags view and enable the Ingestion staging flag.

Important: Where is my incoming data?

Custom Telemetry Data
When this feature is enabled, new custom telemetry object types will not automatically appear in the openmetrics section of the Ingestion->metrics screen.

Instead, new custom telemetry data is available in the staging area for review and curation, so that users fully control and publish how their data is stored and represented in Crosswork Assurance. The object types must be published and metrics must be toggled on before any data will start flowing to Crosswork Assurance; this article walks you through the workflow to get telemetry data flowing into Crosswork Assurance.

Assurance Sensor & IOS XR/IOS XE out-of-the-box default offering data

If you are ingesting data from our Crosswork Assurance Sensors, or if you are using the out-of-the-box IOS XE or IOS XR Telemetry Collection, then you will not need to use this feature to publish new object types. Crosswork Assurance comes with pre-defined, ready-to-use object types that work for Assurance Sensors and the IOS XE and IOS XR default offering. Sessions will appear in the inventory automatically from these default types.

Key takeaway: This feature is only required if you are creating and publishing new custom telemetry object types, or adding additional sensor paths to the out-of-the-box offering for IOS XE and IOS XR routers.

Getting started: Navigate to the landing page

The Ingestion Staging feature main landing page is located in Ingestion -> Object Types, shown below.

Screenshot:

The landing page for the ingestion staging feature (shown above) displays a table with 4 columns:

• Display Name
• Object Type
• Status
• # of metrics

Status values are as follows:

• Published - Object type and metrics are published in an ingestion specification. This object type is ready for use in Crosswork Assurance.
• Drafted - Object type and metrics have been manually drafted using the ingestion staging feature, but the ingestion specification is still in draft state. This object type is not ready for use in Crosswork Assurance until it is published.
• Discovered - incoming data flow from a sensor collector has resulted in this newly discovered object type, however it has not yet been curated and published in an ingestion specification in Crosswork Assurance. This status means it is ready for editing in the ingestion staging feature. This object type is not ready for use in Crosswork Assurance until the ingestion specification is created and published.
• Unpublished Changes - This refers to a previously published object type that has been modified, but the latest edits/changes have not yet been published. It differs from the draft state because unlike a "draft" (which does not have a first published version), this state refers to published object types that have been modified, and the latest edits/changes that have not yet been published.

Setting up new data sources in Crosswork Assurance

The most common workflow for this feature is to ingest telemetry from a data source like a router or a message bus with a custom-built Telemetry Collector. Once the ingestion components (Telemetry Collector and Sensor Collector) are sending data to Crosswork Assurance, the new object types will appear in the list with the status discovered. From this state, users open the ingestion staging feature to edit and curate their incoming data, and specify exactly how they want it to appear in Crosswork Assurance.

The typical workflow involves 4 steps:

1. Ingest custom telemetry into Crosswork Assurance

In this step, CX or Partners set up custom ingestion pipelines to bring telemetry into PCA. This involves customizing a Telemetry Collector and a Sensor Collector to bring in the desired telemetry from the data source (router, message bus) and securely transmit the data to Crosswork Assurance.

Refer to the instructions on how to customize the out-of-the-box IOS XE and IOS XR defaults if you want to explore customizing Telegraf to bring in additional sensor path metrics from an IOS XE or IOS XR router.

Alternatively, if you have your own Telegraf configuration file and are ready to use it with Crosswork Assurance, you can upload it to build a custom Telemetry Collector following these instructions.

2. Review & select which metrics and dynamic metadata to ingest (left column)

Once you have data coming from your telemetry source to Crosswork Assurance via a Telemetry Collector and a Sensor Collector, you are ready to use this feature.

• Navigate to the landing page and select the Discovered object type from the list. This object type name will correspond to the name used in the telegraf configuration of the Telemetry Collector.
• You will be presented with a Create pop-up window. You must enter a category. Enter the vendor (like "Cisco" or "Ciena") that represents the telemetry data source. Screenshot below.
  

Upon entering a vendor in the category box and selecting the checkmark to save, you are presented with the main Ingestion Staging editor:

The collected data in the left column displays the incoming data from Sensor Collector.

Ensure the Collected data in the left side column is what you intended to ingest from the data source. If it is not correct, you will require changes in the telemetry collector configuration to tweak the data/metrics you are collecting.

• In the Unmapped bottom section of the left column (shown below), use the + button to confirm that you want to ingest that data (metric or dynamic metadata) into Crosswork Assurance.

• The + action automatically moves the field up to either the metric or dynamic metadata category above. Numeric values go to metrics and text values go to Dynamic metadata.

• A line appears between the left column value and right column value (screenshot below). This action has automatically mapped the value from the Collected data to the Normalized data (right column). This line is a visual indicator that you intend to ingest the data into Crosswork Assurance.
  

• Repeat for all unmapped data. If you want to ingest the data item into Crosswork Assurance, it needs to be mapped using the + button.

3. Curate and normalize your data

Use the middle column to edit each metric.

Click on the chevron > to open the side panel (screenshot below) and edit the values to describe how you want them expressed in Crosswork Assurance:

• Name
• Units
• Direction (NON=None; SD = Source to Destination; DS = Destination to Source ; RT = Round Trip)
• Formula - SQL formulas are allowed; be careful to express the variables with single quotes

Repeat this for all metrics and dynamic metadata.

4. Publish and enable data flow

Once satisfied with all the changes in the Normalized data column, select Publish in the top right corner (screenshot below).

For the final step, go to the Ingestion -> Metrics screen, select your object type, and toggle on all the metrics one by one to start data flowing into Crosswork.

Helpful workflow: Importing & exporting ingestion specifications between Crosswork Assurance instances

Users can save time and duplicate effort by exporting the JSON format Ingestion Specification for a specific object type and importing the specification into a different instance of Crosswork Assurance.

This can be useful if you have defined and published an object type and metrics on a test environment, and want to export the ingestion specification from the test environment and import it into a different Crosswork Assurance instance/environment, thus saving the time and effort of rewriting and publishing the specification from scratch.

The Export button is available from the top right corner of the Ingestion Staging editor screen, triggering the download of the JSON file ingestion specification to your local computer:

The Import function is a file upload option presented on the Create workflow, located on the top right corner of the landing page:

A pop-up is provided to enable upload of your ingestion specification: