Key-Value

The Key Value Parser in Observo AI allows users to convert log data that is in key-value format into a structured JSON format.

Purpose

The purpose of the Key Value Parser is to efficiently extract, normalize, and structure semi-structured log data for downstream processing, analytics, and storage. This enhances observability, security monitoring, and automation in log-driven workflows.

In a key-value log, each log entry typically consists of a series of key-value pairs, where a key represents the attribute or property, and the associated value holds the corresponding data or information. For instance, a key-value log entry might look like this:

timestamp=2023-09-29T12:00:00 event_type=error message=An error occurred

In this example, the log entry contains three key-value pairs: timestamp, event_type, and message. Each key is associated with a value.

A key-value parser is employed to perform the following tasks:

  • Identification: It recognizes and separates key-value pairs within each log entry. This typically involves identifying the key and value portions of the data.

  • Extraction: It extracts the keys and their corresponding values from the log entry, isolating them for further processing.

  • Structuring: The extracted key-value pairs are then organized into a structured JSON format. JSON is a widely-used data interchange format that provides a hierarchical and organized way to represent data.

For example, the key-value log entry mentioned earlier could be transformed into JSON format like this:

{
   "timestamp": "2023-09-29T12:00:00",
   "event_type": "error",
   "message": "An error occurred"
}

This structured JSON format makes it easier to analyze, query, and work with the log data, as it organizes information in a well-defined manner.

Keys and values can be wrapped with ". " characters can be escaped using "\."

Usage

Select Key Value Parser transform. Add Name (required) and Description (optional).

General Configuration:

  • Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.

  • Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

Key Value Parser: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.

Fields to Parse Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs:

  • Field Name: Specify the field name whose value will be parsed using the Key-Value parser.

  • New Field Name: Specify the destination field name for storing the parsed data. If left empty, the parsed data will replace the root level. Existing root level fields with matching names will be replaced.

  • Key-Value Delimiter: The string that separates keys from values. For example: "@timestamp="Sun Jan 10 16:47:39 EST 2021" level=info msg="Stopping all fetchers". Whitespaces around delimiters are ignored.

  • Field Delimiter: The string that separates each key/value pair. For example: "@timestamp="Sun Jan 10 16:47:39 EST 2021", level=info, msg="Stopping all fetchers". Whitespaces around delimiters are ignored. The Default is space/s.

Examples

Remap Key Values

Scenario: Remap two (2) existing field names to two (2) new ones.

Fields to Parse Rules

  • Rule1:

Field Name
New Field Name
Key-Value Delimiter
Field Delimiter

log.message

parsed_data

"="

", "

  • Rule2:

Field Name
New Field Name
Key-Value Delimiter
Field Delimiter

metadata.attributes

metadata.parsed

":"

"; "

Input

{
  "timestamp": "2024-02-15T10:30:00Z",
  "message": "level=INFO, status=success, request_id=\"abc-123\", response_time=150ms, user=\"john.doe\"",
  "metadata": "server:prod-1; region:us-west; service:api; instance_type:t2.micro"
}

Output

{
  "metadata": {
    "parsed": {
      "instance_type": "t2.micro",
      "region": "us-west",
      "server": "prod-1",
      "service": "api"
    }
  },
  "parsed_data": {
    "level": "INFO",
    "request_id": "abc-123",
    "response_time": "150ms",
    "status": "success",
    "user": "john.doe"
  },
  "timestamp": "2024-02-15T10:30:00Z"
}

Results: This ensures that the log data uses standardized field names that are more descriptive and consistent for downstream processing, such as SIEM ingestion and further analysis.

Key-Value Parser Best Practices

When utilizing the Key Value Parser in Observo AI pipelines for SIEM ingestion, the goal is to extract structured insights from logs while leveraging AI-powered automation, pattern recognition, and sentiment analysis. Observo AI follows these best practices:

  1. Structured Parsing & Normalization

    • Handles various key-value formats (e.g., key=value, key: value, JSON) to ensure consistent parsing.

    • Maps extracted keys to SIEM-compatible field names (e.g., rename src_ip to source_ip).

    • Leverages Observo AI ML models to intelligently extract and normalize nested or misformatted key-value pairs.

    • Standardizes data types (timestamps, IP addresses, numeric values) for effective correlation.

  2. Advanced Machine Learning & Pattern Recognition

    • Employs ML-driven pattern recognition to dynamically detect custom key-value structures.

    • Integrates sentiment analysis to classify log messages by positive, neutral, or negative sentiment for alert prioritization.

    • Analyzes tag trends for recurring patterns, enabling identification of anomalies and critical security events.

  3. Log Data Summarization & Insights

    • Aggregates log data based on key identifiers (user IDs, device names, IP addresses) to generate concise summaries.

    • Provides data insights through automated dashboards that display log summaries by keys and track tag trends over time.

    • Uses summarization to reduce noise by grouping similar log events, improving SIEM performance and reducing storage costs.

  4. Efficient SIEM Ingestion & Automation

    • Preprocesses parsed key-value data into SIEM-friendly formats (JSON) for seamless ingestion.

    • Samples or throttles data to ensure high throughput and prevent SIEM overload.

    • Automates threat scoring using Observo AI ML models to assign risk levels to logs before ingestion, facilitating better alert prioritization.

  5. Continuous Learning & Model Adaptation

    • Retrains AI models periodically to accommodate evolving log formats and data trends.

    • Monitors parser performance through regular analysis of tag trends, pattern frequency, and sentiment shifts.

    • Automates adjustments in retention policies, alert thresholds, and parsing logic based on AI-driven insights to continuously optimize SIEM integration.

By following these best practices, your key-value parser in conjunction with other Observo AI transforms will not only deliver clean, structured, and enriched log data to the SIEM but also leverage Observo AI’s ML models, automation, and pattern recognition to enhance security monitoring, anomaly detection, and predictive analytics.

Related Functions

  • JSON Parser: Parse log events into structured JSON.

  • CEF (Parser): Extracts and normalizes fields from CEF-formatted logs, enabling efficient search, correlation, and analysis in SIEM systems.

PreviousJSONNextMetric Parser

Last updated

Was this helpful?