Metric Parser
The Metric Parser in Observo AI allows users to convert events into metric data format for analysis and monitoring purposes.
Purpose
The purpose of the Metric Parser is to extract and normalize metric values that are embedded within the path of the event structure. It identifies the type of metric and applies the appropriate parsing logic to ensure that the data is correctly formatted. For counter metrics, the parser is designed to handle field values that can be either a number or a string, offering flexibility in data representation. For metric types such as gauges or histograms, it enforces that the field value is strictly a number to maintain data integrity. This process ensures that the parsed metric data is accurate, optimized for efficient querying, and suitable for downstream analysis and SIEM ingestion.
Usage
Select Metric Parser transform. Add Name (required) and Description (optional).
General Configuration:
Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.
Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.
Metric Parser: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.
Filter Conditions: Defaults to empty. When set, only events that match the specified condition will be serialized into metric data. If conditions are not specified, all events will be serialized into metrics. Events that do not match the condition will be dropped. Based on AND/OR conditions, "+Rule" or "+Group" buttons.
Metric Configuration Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add Buttons to add more rule fields with the following inputs (optional):
Metric Value Field Name: Select the path within the event structure where the metric value is located. If this field holds a null value, the metric won't be emitted.
Notes: For counters, the field value can be either a number or a string.
For counter metric type, if the value is a number and you wish to use it for incrementing, set "Increment By Value" to true.
Alternatively, if "Increment By Value" is false, the counter will increment by 1.
For metrics types other than counters, such as gauges or histograms, the field value must be a number. In this case, the "Increment By Value" parameter does not affect the metric behavior.
Increment By Value: Defaults to disabled. Toggle on to specify whether the metric value for counter metrics should be incremented by the field value. If set to false, the metric value will be incremented by 1. If set to true, the value is incremented by the value found in the specified metric value field. For other metric types, this field has no effect.
Metric Name: Name of the metric. If left empty, the metric name will be derived from the field name containing the metric value.
Metric Namespace: Namespace of the metric. Optional field.
Metric Type: Select Metric Type. Supported types include counter, gauge, histogram, summary, and set.
Metric Configuration Rules: Specify the tags/dimensions of the metric. Each tag consists of a name and a corresponding field path whose value will be used as the tag value. Please avoid specifying field names with high cardinality values. Click Add Buttons to add rule fields with the following inputs (optional):
Metric Tag Name: Name of the metric tag. This can differ from the field name supplying the tag value.
Examples - host - kubernetes.container.idField Name: Path of the field supplying the tag value.
Examples - log.message - kubernetes.container.id
Examples
Collect Kubernetes Production Metrics
Scenario: Collect CPU usage and network traffic metrics from production Kubernetes site.
Metric Parser
Filter Conditions
AND
kubernetes.namespace
equals
production
Rule1
system.cpu.usage
false
cpu_usage_percent
system_metrics
Gauge
Add Metric Tags (Rule1)
host
host.name
container
kubernetes.container.name
Rule2
network.bytes_sent
true
network_traffic_bytes
network_metrics
Counter
Add Metric Tags (Rule2)
interface
network.interface.name
direction
network.direction
Input
{
"timestamp": "2024-02-15T10:30:00Z",
"kubernetes": {
"namespace": "production",
"container": {
"name": "web-server"
}
},
"host": {
"name": "node-01"
},
"system": {
"cpu": {
"usage": 75.5
}
},
"network": {
"bytes_sent": 1024,
"interface": {
"name": "eth0"
},
"direction": "outbound"
}
}
Output
{
"metrics": [
{
"name": "cpu_usage_percent",
"namespace": "system_metrics",
"type": "gauge",
"value": 75.5,
"tags": {
"host": "node-01",
"container": "web-server"
}
},
{
"name": "network_traffic_bytes",
"namespace": "network_metrics",
"type": "counter",
"value": 1024,
"tags": {
"interface": "eth0",
"direction": "outbound"
}
}
]
}
Results: Returns Kubernetes CPU usage and network traffic metrics in standardized format.
Metric Parser Best Practices
To effectively utilize the Metric Parser in Observo AI pipelines, follow these best practices to ensure accurate field extraction, proper metric type selection, and efficient metric tagging for structured insights:
Field Extraction
Specify Precise Field Paths: Always define the exact path within the event structure for the Metric Value Field Name to accurately extract metric values. Ensure the path points to a field containing a valid value to avoid null outputs, as null values prevent metric emission.
Handle Flexible Counter Values: For counter metrics, leverage the parser’s flexibility to process field values as either numbers or strings. Use this capability when dealing with diverse data sources where metric values may not be uniformly formatted.
Enforce Numeric Values for Gauges and Histograms: For non-counter metric types such as gauges, histograms, summaries, sets, ensure the Metric Value Field Name points to a field with numeric values to maintain data integrity, as these metric types require strict numeric input.
Metric Type Selection
Choose Appropriate Metric Types: Select the correct Metric Type based on the data’s purpose:
Use counters for incremental metrics, such as event counts or occurrences.
Use gauge for values that fluctuate, like CPU usage or memory levels.
Use histogram for statistical distributions, such as latency measurements.
Use summary for quantile-based metrics over time.
Use sets for unique value tracking, like unique user IDs.
Utilize Increment By Value for Counters: For counter metrics, toggle Increment By Value to true when the field value represents the increment amount such as incrementing by a specific numeric value. Set to false to increment by 1 for simple event counting, ensuring alignment with the metric’s purpose.
Metric Tagging
Add Relevant Tags for Context: Use Metric Tag Name and Field Name to attach meaningful tags (dimensions) to metrics, such as environment, region, or service name. Tags enhance queryability and correlation in downstream analysis or SIEM systems.
Avoid High Cardinality Tags: Refrain from using fields with high cardinality such as unique IDs, timestamps as tag values to prevent excessive metric dimensionality, which can degrade performance and increase storage costs.
Use Descriptive Tag Names: Ensure Metric Tag Name is clear and distinct from the Field Name supplying the tag value to improve readability and maintainability of the metric configuration.
Configuration Optimization
Leverage Filter Conditions: Enable Filter Conditions to process only relevant events, reducing noise and improving performance. Use AND/OR conditions with +Rule or +Group buttons to create precise filtering logic, ensuring only matching events are serialized into metrics.
Set Meaningful Metric Names and Namespaces: Provide a descriptive Metric Name to clearly identify the metric’s purpose. If left empty, the parser derives the name from the field, but a custom name improves clarity. Use Metric Namespace to group related metrics, enhancing organization for downstream systems.
Bypass When Necessary: Enable Bypass Transform only when specific events should skip metric parsing entirely, ensuring unprocessed events pass through to subsequent transforms without modification or when testing warrants bypass the transform.
Example Application
Scenario Example: When collecting Kubernetes CPU usage and network traffic metrics, configure the Metric Parser with precise field paths such as cpu.usage for gauges, network.bytes for counters, apply filters to target production cluster events, and add tags like cluster=production and node_name. Set Increment By Value appropriately for counters to reflect actual byte counts, ensuring standardized and queryable output.
By adhering to these best practices, the Metric Parser ensures accurate, efficient, and structured metric data extraction, optimized for analysis and SIEM ingestion in Observo AI pipelines.
Related Functions
JSON Parser: Parse log events into structured JSON.
CEF (Parser): Extracts and normalizes fields from CEF-formatted logs, enabling efficient search, correlation, and analysis in SIEM systems.
Last updated
Was this helpful?