CEF
The CEF (Parser) in Observo AI parses the value in CEF (Common Event Format) format. It ignores everything up to the CEF header. Empty values are returned as empty strings. Surrounding quotes are removed from values.
Purpose
The purpose of a CEF (Common Event Format) parser is to extract and normalize log data into a standardized structure that can be easily processed, analyzed, and correlated by security and monitoring tools. It converts raw log entries from various sources into the CEF format, making it compatible with SIEM systems for better event detection and response. By parsing critical fields such as event IDs, timestamps, severity levels, and source/destination information, it enhances visibility, troubleshooting, and security analysis.
Usage
Select CEF Parser transform. Add Name (required) and Description (optional).
General Configuration:
Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.
Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.
Extract: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.
Fields to Parse Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs:
Field Name: Field name on whose value the CEF parser will apply.
New Field Name: Specify the name of the destination field for storing the parsed data. If left empty, the parsed data will be written to the root level. Existing root level fields will be replaced with the new values if their field names match.
Field Name to preserve Syslog: Custom fields refer to additional key-value pairs that can be included in a log event to provide specific and contextual information about an event. These custom fields are not part of the predefined CEF standard fields but are user-defined and can be used to convey extra details that are relevant to the event being logged.
Translate Custom Fields: Defaults to disable. When enabled, these custom fields refer to additional key-value pairs that can be included in a log event to provide specific and contextual information about an event. These custom fields are not part of the predefined CEF standard fields but are user-defined and can be used to convey extra details that are relevant to the event being logged.
Example
Parse AWS VPC Flow Logs with CEF
Scenario: Parse AWS VPC Flow logs and convert them into a CEF format.
Fields to Parse Rules
message
[Empty]
[Empty]
disabled
Input
2 123456789012 eni-abc12345 192.168.1.10 10.0.0.5 443 56321 6 2024-02-27T12:34:56Z 2024-02-27T12:34:57Z 2 200 1234 5678 ACCEPT OK
2 123456789012 eni-def67890 172.31.5.20 198.51.100.10 22 34567 6 2024-02-27T12:35:10Z 2024-02-27T12:35:11Z 1 150 4321 8765 REJECT OK
2 123456789012 eni-ghi90123 10.0.1.15 203.0.113.5 80 54321 6 2024-02-27T12:36:20Z 2024-02-27T12:36:21Z 3 250 567 890 ACCEPT OKOutput
CEF:0|AWS|VPC|2024-02-27T12:34:56Z|200|Flow Log|2|src=192.168.1.10 dst=10.0.0.5 dpt=443 sport=56321 proto=TCP duration=0.001 bytes=200 action=ACCEPT outcome=OK startTime=2024-02-27T12:34:56Z endTime=2024-02-27T12:34:57Z
CEF:0|AWS|VPC|2024-02-27T12:35:10Z|404|Flow Log|2|src=172.31.5.20 dst=198.51.100.10 dpt=22 sport=34567 proto=TCP duration=0.001 bytes=150 action=REJECT outcome=OK startTime=2024-02-27T12:35:10Z endTime=2024-02-27T12:35:11Z
CEF:0|AWS|VPC|2024-02-27T12:36:20Z|500|Flow Log|2|src=10.0.1.15 dst=203.0.113.5 dpt=80 sport=54321 proto=TCP duration=0.001 bytes=250 action=ACCEPT outcome=OK startTime=2024-02-27T12:36:20Z endTime=2024-02-27T12:36:21ZResults: The key benefits of CEF parsing include standardizing log data for easier analysis, enhancing security monitoring, improving event correlation, and increasing visibility into network traffic for faster troubleshooting and incident response.
Best Practices for CEF Parsing
Follow CEF Formatting Standards – Ensure logs strictly adhere to the CEF format (version 0, vendor, product, event type, etc.) to maintain consistency and compatibility with SIEM systems.
Normalize Timestamps – Use UTC timestamps in ISO 8601 format to prevent time discrepancies across different systems.
Map Important Fields – Focus on parsing and including key fields such as source/destination IPs, event severity, action taken, event name, and user information for easier analysis.
Filter Out Noise – Remove irrelevant or duplicate log entries to reduce data storage and improve processing performance.
Preserve Original Log Data – Retain raw logs alongside parsed logs to allow for deeper investigations and forensic analysis when needed.
Align with Security Frameworks – Map CEF fields to common schemas such as CIM (Common Information Model) or security frameworks for easier integration with security tools.
Use Automation – Leverage Observo AI Pipelines to automatically parse and enrich logs, ensuring consistent log quality and processing.
Monitor for Parsing Errors – Continuously validate parsed data to detect and address any discrepancies or missing information.
Leverage Enrichment – Enrich parsed logs with additional metadata such as geolocation, threat intelligence, or contextual details to improve security analysis.
Implement Proper Log Retention – Define log rotation and retention policies based on compliance requirements and storage capacity to maintain efficient log management.
By following these best practices, you can optimize log parsing for easier integration with SIEM systems, enhance security monitoring, and streamline data processing.
Related Functions
Syslog Parser: Parse Syslog event into structured JSON.
AWS VPC Flow Logs Parser: Extract, normalize, and structure network traffic logs from AWS Virtual Private Cloud (VPC) environments.
Last updated
Was this helpful?