Telemetry Ingestion via Node Exporter

This article describes how to configure the Telemetry Collector to scrape host-level metrics from Prometheus Node Exporter endpoints and forward them through the Sensor Collector to Crosswork Assurance. This approach is useful for monitoring the health of the infrastructure that hosts your network components, CPU utilization, memory pressure, disk I/O, and network throughput on Linux servers.

Out-of-the-box vs. Custom Telemetry

Crosswork Assurance provides default Telemetry Collector configurations for IOS-XE and IOS-XR routers. This guide covers Node Exporter as a custom data source that requires a custom Telegraf configuration (TOML). For custom telemetry ingestion engagements, contact your Cisco CX or Partner representative.

The Telemetry Collector embeds Telegraf, which includes the prometheus input plugin. By customizing the Telemetry Collector's Telegraf configuration (TOML), you can configure it to scrape Prometheus-format metrics from Node Exporter endpoints and convert them to OpenMetrics format for ingestion into Crosswork Assurance.

Architecture

1. Node Exporter runs on each Linux host you want to monitor and exposes metrics at http://<host>:9100/metrics in Prometheus exposition format.
2. The Telemetry Collector (Telegraf prometheus input plugin) periodically scrapes the /metrics endpoint.
3. Scraped metrics are tagged with objectType, sessionId, and sessionName, then forwarded through the Gateway Sensor Collector to Crosswork Assurance.

Prerequisites

• A deployed Gateway Sensor Collector with the OpenMetrics Gateway connector enabled.
• Prometheus Node Exporter installed and running on each target Linux host, exposing metrics on port 9100.
• Network connectivity from the Telemetry Collector to each Node Exporter endpoint on port 9100.
• Access to the Crosswork Assurance UI with permissions to create Telemetry Collector configurations.

Step 1: Install Node Exporter

Install Node Exporter on each target host. For example, on a Debian/Ubuntu system:

# Download and install (check https://github.com/prometheus/node_exporter/releases for latest version)
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xvfz node_exporter-1.8.2.linux-amd64.tar.gz
sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/

# Run as a systemd service (recommended for production)
sudo useradd --no-create-home --shell /bin/false node_exporter
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Verify that metrics are accessible:

curl http://localhost:9100/metrics | head -20

Step 2: Design the Telegraf Configuration (TOML)

The Telemetry Collector's runtime behavior is controlled by a Telegraf TOML configuration file. When you create a Telemetry Collector with a Custom Telegraf configuration in the UI, you upload a TOML file that defines:

• Inputs -- which Node Exporter endpoints to scrape
• Processors -- how to build session identity tags for Crosswork Assurance

Important: Do not include an [[outputs.*]] section in the TOML file. The product automatically appends its own output configuration for forwarding data to the Sensor Collector. Including an output section causes a "violation of Telegraf's configuration" error.

Key Design Principles

• The objectType tag is required. The Telemetry Collector agent requires an objectType tag whose value matches the name of the ingestion dictionary that defines the metrics and dimensions being ingested for the objects. Use Telemetry Ingestion Staging to create and manage your ingestion dictionaries.

• Session identity processors are required. The Telemetry Collector opens one reporting stream per sessionId. Include [[processors.template]] blocks that build sessionId and sessionName from your parsed tags.

• Use metric_version = 2 for a column-oriented layout where each Prometheus metric name becomes a Telegraf field key. This is the recommended format for Prometheus-style data.

Example: Node Exporter TOML Configuration

The following example scrapes metrics from one or more Node Exporter endpoints. Add additional URLs to the urls array to monitor multiple hosts.

# [agent]
#   interval = "60s"
#   flush_interval = "20s"
#   debug = true
#   omit_hostname = true

[[inputs.prometheus]]
  urls = ["http://node-exporter:9100/metrics"]
  interval = "60s"

  ## Optional: HTTP Basic Auth or Bearer Token for secured endpoints
  # username = ""
  # password = ""
  # bearer_token_string = ""

  ## Optional: TLS configuration
  # tls_ca = "/path/to/ca.pem"
  # insecure_skip_verify = false

  [[processors.template]]
    order = 1
    namepass = ["node*"]
    tag = "direction"
    template = "-1"

  [[processors.template]]
    order = 2
    namepass = ["node_cpu*"]
    tag = "objectType"
    template = "node-exporter-cpu"
  [[processors.template]]
    order = 2
    namepass = ["node_cpu*"]
    tag = "sessionName"
    template = 'host-{{ .Tag "host" }}_cpu-{{ .Tag "cpu" }}_node-exporter-cpu'
  [[processors.template]]
    order = 2
    namepass = ["node_cpu*"]
    tag = "sessionId"
    template = 'host-{{ .Tag "host" }}_cpu-{{ .Tag "cpu" }}_node-exporter-cpu'

  [[processors.template]]
    order = 2
    namepass = ["node_disk*"]
    tag = "objectType"
    template = "node-exporter-disk"
  [[processors.template]]
    order = 2
    namepass = ["node_disk*"]
    tag = "sessionName"
    template = 'host-{{ .Tag "host" }}_device-{{ .Tag "device" }}_node-exporter-disk'
  [[processors.template]]
    order = 2
    namepass = ["node_disk*"]
    tag = "sessionId"
    template = 'host-{{ .Tag "host" }}_device-{{ .Tag "device" }}_node-exporter-disk'

  [[processors.template]]
    order = 2
    namepass = ["node_filesystem*"]
    tag = "objectType"
    template = "node-exporter-filesystem"
  [[processors.template]]
    order = 2
    namepass = ["node_filesystem*"]
    tag = "sessionName"
    template = 'host-{{ .Tag "host" }}_device-{{ .Tag "device" }}_node-exporter-filesystem'
  [[processors.template]]
    order = 2
    namepass = ["node_filesystem*"]
    tag = "sessionId"
    template = 'host-{{ .Tag "host" }}_device-{{ .Tag "device" }}_node-exporter-filesystem'

  [[processors.template]]
    order = 2
    namepass = ["node_memory*"]
    tag = "objectType"
    template = "node-exporter-memory"
  [[processors.template]]
    order = 2
    namepass = ["node_memory*"]
    tag = "sessionName"
    template = 'host-{{ .Tag "host" }}_node-exporter-memory'
  [[processors.template]]
    order = 2
    namepass = ["node_memory*"]
    tag = "sessionId"
    template = 'host-{{ .Tag "host" }}_node-exporter-memory'


  [[processors.template]]
    order = 2
    namepass = ["node_netstat*"]
    tag = "objectType"
    template = "node-exporter-netstat"
  [[processors.template]]
    order = 2
    namepass = ["node_netstat*"]
    tag = "sessionName"
    template = 'host-{{ .Tag "host" }}_node-exporter-netstat'
  [[processors.template]]
    order = 2
    namepass = ["node_netstat*"]
    tag = "sessionId"
    template = 'host-{{ .Tag "host" }}_node-exporter-netstat'


  [[processors.template]]
    order = 2
    namepass = ["node_network*"]
    tag = "objectType"
    template = "node-exporter-network"
  [[processors.template]]
    order = 2
    namepass = ["node_network*"]
    tag = "sessionName"
    template = 'host-{{ .Tag "host" }}_device-{{ .Tag "device" }}_node-exporter-network'
  [[processors.template]]
    order = 2
    namepass = ["node_network*"]
    tag = "sessionId"
    template = 'host-{{ .Tag "host" }}_device-{{ .Tag "device" }}_node-exporter-network'


  [[processors.starlark]]
  ## 1) filters out any metric missing the objectType label, we dont accept those anyway. 
  ## 2) removes the leading node_xyz part of each metric name because we already encoded 
  ## this information into the objectType so no need to repeat it.  
    order = 3
    namepass = ["node_*"]

    source = '''
def apply(metric):

    if "objectType" not in metric.tags:
        # If the "objectType" tag is missing, drop the metric by returning None
        return None

    parts = metric.name.split("_")
    if len(parts) >= 2: # Ensure there are at least two words to remove
        metric.name = "_".join(parts[2:]) # Join parts from the third word onwards

    return metric
'''


[[processors.tag_limit]]
order = 4
  #fieldinclude = ["*"]
  ## Maximum number of tags to preserve
  limit = 8

  ## List of tags to preferentially preserve
  keep = ["sessionId", "sessionName", "objectType", "direction"]


# [[outputs.file]]  
#    data_format = "prometheus"
#    #data_format = "json"
 
#    ## Files to write to, "stdout" is a specially handled file.
#    files = ["stdout","/tmp/metrics.out"]

Security note: Use environment variables for the HTTP credentials rather than hardcoding them in the TOML file. Telegraf supports secret stores for managing sensitive values.

Tip: If you want to extract just the hostname from the URL for cleaner session names, use the Telegraf regex processor to transform the url tag before the template processors run.

Key Node Exporter Metrics

Node Exporter exposes hundreds of metrics. The following are commonly used for infrastructure monitoring:

Prometheus Input Plugin Reference

For the complete list of configuration options (including Kubernetes service discovery, Consul integration, and TLS settings), see the Telegraf Prometheus input plugin documentation.

Step 3: Create the Telemetry Collector in the UI

For general instructions on creating and managing Telemetry Collector configurations, see Configure Telemetry Collector. The steps below are specific to uploading a custom Node Exporter TOML configuration.

1. In the Crosswork Assurance UI, navigate to Sensors > Collectors.
2. Click the + button to add a new collector.
3. In the New collector dialog, select Telemetry and click Configure collector.
4. Enter a Name for the Telemetry Collector (e.g., Node_Exporter_Collector).
5. Set the Telegraf Configuration dropdown to Custom.
6. In the Custom Telegraf.conf section, click the upload icon and select your TOML file.
7. Enter the Sensor Collector Data Address -- the hostname or IP of your Gateway Sensor Collector.
8. Enter the Sensor Collector Data Port -- the data port of your Gateway Sensor Collector (e.g., 39888).
9. Configure SSL options as needed:
   • Use SSL communication -- enable for encrypted connections to the Sensor Collector
   • Allow self-signed certificates -- enable if the Sensor Collector uses self-signed certificates
   • Skip SSL hostname check -- enable if the hostname does not match the certificate
10. Click the checkmark (save) button.

Generate the Auth Token

After saving the Telemetry Collector configuration:

1. On the Summary tab, click Generate auth token.
2. Copy the Agent secret (YAML) content. This contains the agentId and authenticationToken that the Telemetry Collector container needs to authenticate with Crosswork Assurance.
3. Save this content as secrets.yaml in your deployment directory.

Important: The auth token is displayed only once. Copy and save it immediately.

Alternative: API Method (Base64)

You can also create the Telemetry Collector configuration via API by Base64-encoding the TOML file and submitting it as the customTransformationConfig field.

1. Base64-encode your TOML file:

   base64 -w 0 my-config.toml > config_base64.txt
   

2. Use the Agent Configuration API to create the Telemetry Collector. For detailed API instructions, see Telemetry Collector Configuration - MDT Dial-Out (alternate API method).

Step 4: Deploy the Telemetry Collector

Deploy the Telemetry Collector container using the standard procedure for your environment:

• Docker: See Deploy Telemetry Collector in Docker
• Kubernetes: See Deploy Telemetry Collector in Kubernetes

The Telemetry Collector connects to the Sensor Collector, receives its custom TOML configuration, and begins scraping the configured Node Exporter endpoints.

Note: Ensure the Telemetry Collector container has network access to each Node Exporter endpoint on port 9100. For Docker deployments, the container must be on a network that can reach the target hosts. For Kubernetes deployments, verify that the Node Exporter service is accessible from the Telemetry Collector pod's namespace.

Step 5: Verify

1. Check the Telemetry Collector logs for successful startup:

   Config watcher started
   Loading config: /etc/Telemetry/Telemetry.conf
   

2. Confirm that metrics appear in the Crosswork Assurance UI under the expected monitored objects.
3. To enable debug logging for troubleshooting:

   docker exec -it <container_id> sh
   agentstatus telemetry datapathLogging true
   exit
   docker logs <container_id> -f
   

4. Test Node Exporter connectivity independently:

   curl http://<node-exporter-host>:9100/metrics | head -20
   

Troubleshooting

Related Topics

• Configure Telemetry Collector
• Deploy Telemetry Collector in Docker
• Deploy Telemetry Collector in Kubernetes
• Telemetry Ingestion Solution Architecture
• Telemetry Collector Configuration - MDT Dial-Out (alternate API method)
• Enable and Configure Telemetry Ingestion Staging
• Telemetry Ingestion with Kafka
• Telemetry Ingestion with SNMP