JSON

The JSON Parser in Observo AI parses log events into structured JSON.

Purpose

The JSON parser enables the transformation of embedded JSON within a string field into a well-structured JSON (JavaScript Object Notation) format. Log entries frequently include fields containing embedded JSON structures.

For example, consider a log entry with the following embedded JSON structure within a field value:

{
   "user_info": "{\"name\":\"John Doe\",\"email\":\"[email protected]\"}",
   "event": "Login",
   "timestamp": "2023-09-29T12:00:00"
}

In this example, the JSON Parser can be used to parse this into structured events. The output after applying the JSON Parser to the provided input will look like this:

{
   "user_info": {
       "name": "John Doe",
       "email": "[email protected]"
   },
   "event": "Login",
   "timestamp": "2023-09-29T12:00:00"
}

Usage

Select JSON Parser transform. Add Name (required) and Description (optional).

General Configuration:

• Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.

• Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

JSON Parser: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.

Fields to Parse Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs:

• Field Name: Field name on whose value the JSON parser will apply.

• New Field Name: New field name for storing the parsed JSON structure. If left blank, the JSON will be stored at the root level.

• Explode Array Events: If true and the JSON field is an array, explode the array into multiple events after parsing.

Examples

Simple JSON Parsing

Scenario: Parse user_info into JSON format.

Fields to Parse Rules

Field Name	New Field Name	Explode Array Events
user_info	[Empty]	disabled

Input

{
  "user_info": "{\"name\":\"John Doe\",\"email\":\"[email protected]\"}",
  "event": "Login",
  "timestamp": "2023-09-29T12:00:00"
}

Output

{
  "name": "John Doe",
  "email": "[email protected]"
  "event": "Login",
  "timestamp": "2023-09-29T12:00:00"
}

Results: The parsed data will be stored at root level.

Nested JSON Parsing

Scenario: Parse user_info into JSON format.

Fields to Parse Rules

Field Name	New Field Name	Explode Array Events
user_info	user_info	disabled

Input

{
  "user_info": "{\"name\":\"John Doe\",\"email\":\"[email protected]\"}",
  "event": "Login",
  "timestamp": "2023-09-29T12:00:00"
}

Output

{
  "user_info": {
      "name": "John Doe",
      "email": "[email protected]"
  },
  "event": "Login",
  "timestamp": "2023-09-29T12:00:00"
}

Results: The parsed data will be stored in the same field, overwriting the original content.

JSON with Array

Scenario: Explode events within an array to two (2) separate events. Fields to Parse Rules

Field Name	New Field Name	Explode Array Events
user_info	user_info	enabled

Input

[{"user_info": {"name": "John Doe", "email": "[email protected]"}, "event": "Login", "timestamp": "2023-09-29T12:00:00"},
{"user_info": {"name": "Jane Smith", "email": "[email protected]"}, "event": "Logout", "timestamp": "2023-09-29T12:10:00"}]

Output (After JSON Exploding) Event 1:

{
 "user_info": {
   "name": "John Doe",
   "email": "[email protected]"
 },
 "event": "Login",
 "timestamp": "2023-09-29T12:00:00"
}

Event 2:

{
 "user_info": {
   "name": "Jane Smith",
   "email": "[email protected]"
 },
 "event": "Logout",
 "timestamp": "2023-09-29T12:10:00"
}

Results: Parsed data will be stored in the same field, overwriting the original content.

Best Practices for JSON Parsing in SIEM

1. Ensure JSON Validity Before Ingestion – Validate JSON logs before sending them to SIEM tools to avoid ingestion failures due to malformed data.

2. Flatten Nested JSON Fields – Convert deeply nested structures into key-value pairs where possible to improve search performance and compatibility with SIEM query languages.

3. Leverage Indexing for Key Fields – Identify and index critical fields like timestamp, user, IP address, event type, and severity to optimize search speed and correlation.

4. Standardize Timestamp Formats – Convert timestamps to a consistent format (e.g., ISO 8601 or UNIX epoch) and ensure proper time zone handling to maintain accurate event sequencing.

5. Filter Unnecessary Fields – Reduce storage costs and improve query efficiency by filtering out irrelevant or redundant fields before indexing.

6. Normalize Field Naming Conventions – Use consistent naming conventions (e.g., camelCase vs. snake_case) across different log sources to simplify field mapping and correlation.

7. Implement Schema Validation – Define a schema for expected JSON structures to prevent unexpected field changes from breaking queries and dashboards.

8. Avoid Large JSON Payloads in a Single Event – Break large JSON logs into smaller, manageable events to prevent performance issues and improve searchability.

9. Use JSON Normalization for Correlation – Normalize key fields across different data sources to enable effective correlation of security events (e.g., mapping source_ip and destination_ip consistently).

10. Monitor and Optimize JSON Parsing Performance – Continuously track parsing and query performance, making adjustments to field extractions, indexing, and storage settings as needed.

By following these best practices, JSON logs are efficiently parsed, indexed, and analyzed within SIEM tools like Splunk and Elasticsearch, ensuring better performance and more effective security monitoring.

Related Functions

• Syslog Parser: Parse Syslog event into structured JSON.

• CEF Parser: Extracts and normalizes fields from CEF-formatted logs, enabling efficient search, correlation, and analysis in SIEM systems.