Grok

The Grok Parser in Observo AI defines the grok patterns required for parsing the log event.

Purpose

The purpose of the Grok parser is to parse and structure unstructured log data by matching it against predefined patterns, enabling easier extraction of key information and transforming logs into a more readable and queryable format. It simplifies the process of extracting useful fields such as IP addresses, timestamps, error codes, and event messages from complex log entries, making it easier to analyze and visualize data in monitoring tools.

Usage

Select Grok Parser transform. Add Name (required) and Description (optional).

General Configuration:

• Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.

• Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

Grok Parser: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.

Groks Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs:

• Field Name: Field name on whose value the grok patterns will apply. The Grok patterns are tried in order until the first match.

• New Field Name: Specify the name of the destination field for storing the parsed data. If left empty, the parsed data will be written to the root level. Existing root level fields will be replaced with the new values if their field names match.

• Grok Patterns: The array of Grok patterns used to parse the event data. You can provide multiple Grok patterns, and the first one that successfully matches will be utilized for parsing the event. Simply click the Add button to add a Grok Pattern. Add as many Grok Patterns as required. To help you define these patterns effectively, you can make use of these resources:

• Grok Basics Grok Debugger Grok Patterns

• Example: %{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel}: User '%{USERNAME:username}' successfully logged in from IP '%{IP:ip}'

Examples

Grok Windows Security Logs

Scenario: Parse Windows Security logs and convert them into a Grok format.

Fields to Parse Rules

Field Name	New Field Name	Field Name to preserve Syslog	Translate Custom Fields
message	[Empty]	[Empty]	disabled

Input

EventCode=4624 EventType=Information TimeGenerated=2024-02-27T12:34:56Z UserName=Administrator SourceNetworkAddress=192.168.1.10 WorkstationName=Workstation01 LogonType=2 LogonProcess=User32 AuthenticationPackage=Negotiate
EventCode=4634 EventType=Information TimeGenerated=2024-02-27T12:35:10Z UserName=Administrator SourceNetworkAddress=192.168.1.10 WorkstationName=Workstation01 LogoffType=2 LogoffProcess=User32
EventCode=4672 EventType=Information TimeGenerated=2024-02-27T12:36:00Z UserName=Admin SourceNetworkAddress=192.168.1.10 WorkstationName=Workstation02 SpecialPrivileges=SeDebugPrivilege SeTakeOwnershipPrivilege

Grok Pattern for Windows Security Logs

EventCode=%{INT:event_code} EventType=%{WORD:event_type} TimeGenerated=%{TIMESTAMP_ISO8601:timestamp} UserName=%{USERNAME:user_name} SourceNetworkAddress=%{IPV4:source_ip} WorkstationName=%{WORD:workstation_name} (LogonType|LogoffType|SpecialPrivileges)=%{DATA:logon_type} (LogonProcess|LogoffProcess|SeDebugPrivilege|SeTakeOwnershipPrivilege)=%{DATA:log_process}

Output

[
  {
    "event_code": 4624,
    "event_type": "Information",
    "timestamp": "2024-02-27T12:34:56Z",
    "user_name": "Administrator",
    "source_ip": "192.168.1.10",
    "workstation_name": "Workstation01",
    "logon_type": "2",
    "log_process": "User32"
  },
  {
    "event_code": 4634,
    "event_type": "Information",
    "timestamp": "2024-02-27T12:35:10Z",
    "user_name": "Administrator",
    "source_ip": "192.168.1.10",
    "workstation_name": "Workstation01",
    "logon_type": "2",
    "log_process": "User32"
  },
  {
    "event_code": 4672,
    "event_type": "Information",
    "timestamp": "2024-02-27T12:36:00Z",
    "user_name": "Admin",
    "source_ip": "192.168.1.10",
    "workstation_name": "Workstation02",
    "logon_type": "SeDebugPrivilege SeTakeOwnershipPrivilege",
    "log_process": ""
  }
]

Results: The key benefits of Grok parsing include transforming unstructured log data into structured, queryable formats, enabling easier analysis, improved security monitoring, and faster troubleshooting.

Best Practices for Grok Parsing with SIEMs

1. Align with SIEM's Data Model – Ensure that Grok parsing aligns with the SIEM’s data model (e.g., Common Information Model in Splunk or Elastic Common Schema in Elasticsearch) to make it easier to correlate and analyze data across different log sources.

2. Optimize for Performance – Use efficient regular expressions in Grok patterns to avoid slowing down the SIEM’s indexing and search capabilities, especially when dealing with large log volumes.

3. Prioritize Key Security Fields – Focus on parsing critical fields such as timestamp, source/destination IP, event type, user, action taken, and status for effective security monitoring and incident response.

4. Leverage Custom Patterns – Create custom Grok patterns for your unique log formats or specific use cases, ensuring better extraction and structure of log data.

5. Normalize Data for Consistency – Normalize fields such as timestamps, IP addresses, and event names to a standard format across all log sources to improve correlation and analysis within the SIEM.

6. Validate and Test Parsing Regularly – Continuously test and validate Grok patterns with real-world log samples to ensure accurate and consistent parsing, and adjust patterns as needed for evolving log formats.

7. Filter Out Unnecessary Data – Implement filters to remove irrelevant or redundant log entries, reducing the size of your dataset, optimizing storage, and focusing analysis on high-priority security events.

8. Enrich Data During Parsing – Use the Grok parser to enrich logs with additional context (such as geolocation, threat intelligence, or device type) to improve security insights and correlation.

9. Handle Multi-Line and Complex Logs Properly – For logs that span multiple lines (e.g., stack traces, large event logs), ensure proper handling so that the log entries are parsed as single events to maintain their context.

10. Document Parsing Rules – Maintain thorough documentation of your Grok patterns and field extraction rules, especially when custom patterns are used, so that teams can quickly adapt or troubleshoot as needed.

11. Monitor Grok Parsing Performance – Regularly monitor the performance of Grok parsing within your SIEM, especially when scaling to larger data sets, to ensure smooth log ingestion and minimize delays.

By following these best practices, you ensure that Grok parsing within your SIEM is efficient, accurate, and helps to extract the most meaningful insights from your logs, improving overall security monitoring and incident response.

Related Functions

• Syslog Parser: Parse Syslog event into structured JSON.

• CEF Parser: Extracts and normalizes fields from CEF-formatted logs, enabling efficient search, correlation, and analysis in SIEM systems.