Enriching Data

Structuring Logs

Use Case: Handling unstructured logs, which can be entirely raw or may contain a mix of unstructured and structured data elements.

Example: Imagine you have log data from a web server where each log entry is a combination of plain text, timestamps, and JSON objects. For instance:

[2023-01-15 12:45:00] - User login successful: {"user": "john_doe", "status": "success"}

Transforms:

• Grok: Use Grok patterns to extract structured data from unstructured events.

• JSON Parsing: Parse JSON objects within the events.

• CEF: Extract structured data from Common Event Format (CEF) events.

• Key Value: Extract structured data from logs in simple key-value format.

• Syslog: Parse Syslog events to a structured format.

Log Event Modification

Use Case: Modifying log events to improve consistency or add meaningful information.

Example: You may need to rename fields for consistency, flatten nested fields, or derive new fields to provide additional context. For example, renaming a field from "source_ip" to "client_ip."

Transforms:

• Rename Field: Change the name of a field.

• Numerify: Convert field values to numeric format.

Event Enrichment

Use Case: Enhancing events with additional metadata or derived fields.

Example: Adding metadata such as "service name" or "app name" to log events for improved context.

Transforms:

• Add Field: Insert new fields with relevant information.

• Extract Regex: Extract information using regex patterns from log entries.

Field Encoding/Decoding

Use Case: Encoding or decoding field values within events, often required for data transformation.

Example: Encoding binary data as Base64 or decoding URL-encoded data within log entries.

Transforms:

• Encode (e.g., Base64 encoding, URL encoding)

• Decode (e.g., URL decoding)

Sentiment-Based Classification

Use Case: Classifying events based on sentiment analysis.

Transforms:

• Apply sentiment analysis to log messages to determine the emotional tone or sentiment.