CSV

The CSV Parser in Observo AI parses fields containing CSV (Comma-Separated Values) data into structured objects. It supports custom delimiters, both dynamic and static headers, and outputs the parsed data into a specified field.

Purpose

The purpose of a CSV parser is to extract and normalize CSV-encoded data into a structured JSON-like format that can be easily processed by downstream components. This enables better analysis, enrichment, and transformation of event data. By converting raw CSV strings into key-value mappings with well-defined headers, the parser ensures consistent data handling, supports flexible data ingestion, and makes CSV-based log data compatible with analytics pipelines.

Usage

Select CSV Parser transform. Add Name (required) and Description (optional).

General Configuration

• Bypass Transform: Defaults to disabled. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.

• Add Filter Conditions: Defaults to disabled. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

CSV Parser:

Select CSV Parser transform. Add Name (required) and Description (optional).

CSV Fields to Parse: Configure one or more fields to parse as CSV data. First field entries added by default. Click Add button to add new field, with the following inputs:

• CSV Field: Specify the field containing the CSV data to parse. Examples: attributes.csv, log.csv_data.

• Headers Mode: Defines how headers are determined.

  Options	Description
  Dynamic (from field)	Headers are read from another field (Headers Field)
  Static (manual entry)	Headers are manually specified as a comma-separated string (Static Headers).

• Headers Field: Specify the field containing the CSV headers. Example: attributes.csv_headers.

• CSV Delimiter (Optional): The delimiter character for the CSV values. Defaults to ',' if not specified. Can only be one character. Examples: ;, |, :.

• Output Field: Specify the field where the parsed output will be stored. Examples: attributes.parsed, log.parsed_csv.

• Header Delimiter (Optional): The delimiter character for the CSV headers. Defaults to ',' if not specified. Can only be one character. Examples: ,;:

Example

Parse CSV Log Data into Structured Output

Scenario: Parse a CSV field containing user session logs with dynamic headers provided in another field.

CSV Fields to Parse

CSV Field	Headers Mode	Headers Field	CSV Delimiter	Output Field	Header Delimiter
log.csv_data	Dynamic	log.csv_headers	,	log.parsed_csv	,

Input

{
  "log": {
    "csv_headers": "user_id,session_id,timestamp,status",
    "csv_data": "42,abc123,2024-02-27T12:34:56Z,active"
  }
}

Output

{
  "log": {
    "csv_headers": "user_id,session_id,timestamp,status",
    "csv_data": "42,abc123,2024-02-27T12:34:56Z,active",
    "parsed_csv": {
      "user_id": "42",
      "session_id": "abc123",
      "timestamp": "2024-02-27T12:34:56Z",
      "status": "active"
    }
  }
}

Results: CSV strings are transformed into structured objects for easier processing and analysis.

Best Practices for CSV Parsing

1. Validate Headers – Ensure headers accurately represent the CSV data to prevent misaligned fields.

2. Use Consistent Delimiters – Standardize delimiter usage across different data sources to avoid parsing errors.

3. Leverage Dynamic Headers for Flexibility – Use dynamic mode when header definitions may vary by event.

4. Prefer Static Headers for Stability – Use static mode when structure is predictable for consistent parsing.

5. Preserve Raw Data – Keep the original CSV string for troubleshooting and reprocessing if needed.

6. Handle Multi-Character Delimiters Carefully – Test custom delimiters (::, |, etc.) to ensure correct splitting.

7. Avoid Excessively Wide CSVs – Limit the number of columns where possible to reduce processing overhead.

8. Normalize Field Names – Use consistent casing and naming conventions for parsed headers.

9. Monitor for Parsing Errors – Validate output fields to ensure alignment with expected structure.

10. Combine with Enrichment – After parsing, enrich structured data with metadata to improve downstream analytics.

Related Functions

• Syslog Parser: Parse Syslog event into structured JSON.

• CEF Parser: Parse Common Event Format data into structured fields.

• JSON Parser: Parse JSON-encoded strings into structured fields.