Syslog

The Syslog Parser in Observo AI allows users to parse Syslog events into structured JSON.

Purpose

The Syslog parser makes a best effort to parse the various Syslog formats that exist out in the wild. This includes RFC 6587, RFC 5424, RFC 3164, and other common variations (such as the Nginx Syslog style).

Usage

Select Syslog Parser transform. Add Name (required) and Description (optional).

General Configuration:

  • Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.

  • Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

Syslog Parser: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.

Fields to Parse Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs:

  • Field Name: Field name on whose value the Syslog parser will apply.

  • New Field Name: Specify the name of the destination field for storing the parsed data. If left empty, the parsed data will be written to the root level. Existing root level fields will be replaced with the new values if their field names match.

Examples

Parse Syslog Data to New Field

Scenario: Parse Syslog data to a new field.

Fields to Parse Rules

Field Name
New Field Name

message

parsed_syslog

Input

{"timestamp": "2023-08-16T12:05:30Z", "message": "<14>Aug 16 12:05:30 localhost app: Error - Connection timeout"}
{"timestamp": "2023-08-16T12:10:45Z", "message": "<13>Aug 16 12:10:45 localhost web: Info - Request received"}
{"timestamp": "2023-08-16T12:15:20Z", "message": "<18>Aug 16 12:15:20 localhost db: Warning - Slow query"}

Output

{"timestamp": "2023-08-16T12:05:30Z", "message": "<14>Aug 16 12:05:30 localhost app: Error - Connection timeout", "parsed_syslog": {"priority": 14, "timestamp": "Aug 16 12:05:30", "hostname": "localhost", "app": "app", "severity": "Error", "message": "Connection timeout"}}
{"timestamp": "2023-08-16T12:10:45Z", "message": "<13>Aug 16 12:10:45 localhost web: Info - Request received", "parsed_syslog": {"priority": 13, "timestamp": "Aug 16 12:10:45", "hostname": "localhost", "app": "web", "severity": "Info", "message": "Request received"}}
{"timestamp": "2023-08-16T12:15:20Z", "message": "<18>Aug 16 12:15:20 localhost db: Warning - Slow query", "parsed_syslog": {"priority": 18, "timestamp": "Aug 16 12:15:20", "hostname": "localhost", "app": "db", "severity": "Warning", "message": "Slow query"}}

Results: The Syslog data in the "message" field is parsed, and the results are stored in the new field "parsed_syslog."

Parse Syslog Data to Root Level

Scenario: Parse Syslog data to the root level.

Fields to Parse Rules

Field Name
New Field Name

log_data

[Empty]

Input

{"timestamp": "2023-08-16T12:20:00Z", "log_data": "<30>Aug 16 12:20:00 localhost sys: Debug - System message"}
{"timestamp": "2023-08-16T12:25:15Z", "log_data": "<22>Aug 16 12:25:15 localhost app: Warning - Application issue"}
{"timestamp": "2023-08-16T12:30:45Z", "log_data": "<16>Aug 16 12:30:45 localhost audit: Info - User login"}

Output

{"timestamp": "2023-08-16T12:20:00Z", "log_data": {"priority": 30, "timestamp": "Aug 16 12:20:00", "hostname": "localhost", "app": "sys", "severity": "Debug", "message": "System message"}}
{"timestamp": "2023-08-16T12:25:15Z", "log_data": {"priority": 22, "timestamp": "Aug 16 12:25:15", "hostname": "localhost", "app": "app", "severity": "Warning", "message": "Application issue"}}
{"timestamp": "2023-08-16T12:30:45Z", "log_data": {"priority": 16, "timestamp": "Aug 16 12:30:45", "hostname": "localhost", "app": "audit", "severity": "Info", "message": "User login"}}

Results: The Syslog data in the "log_data" field is parsed into the appropriate fields.

Best Practices for Syslog Parser (SIEM)

When utilizing a Syslog Parser for Observo AI pipelines with a SIEM as the destination, you’re assured of structured, efficient, and secure log processing. Below are some best practices:

  • Structured Parsing & Normalization

    • Utilize structured syslog formats (e.g., RFC 5424) to ensure consistent field extraction and normalization (JSON, key-value pairs) for SIEM ingestion.

    • Leverage Observo AI ML models to automatically parse logs, extract critical fields (timestamps, hostnames, severity, event IDs), and apply pattern recognition and sentiment analysis for advanced metadata tagging.

    • Map parsed fields to SIEM schemas (JSON) while tagging logs with sentiment scores to prioritize alerts.

  • Data Enrichment, Summary & Insights

    • Enrich log data in real time by integrating third-party context (Geo-IP, threat intelligence) using advanced machine learning and automation techniques.

    • Generate log data summaries by keys and analyze tag trends to detect recurring patterns and anomalies, enabling proactive detection of security incidents.

    • Use automated data insights to provide a “Log Data Summary” and trend analysis on tags and patterns, helping correlate events and reduce noise before SIEM ingestion.

  • Automation, Scalability & SIEM Integration

    • Implement automated batch processing and smart routing to continuously deliver optimized log streams to SIEM systems, ensuring scalability and high throughput.

    • Integrate advanced ML algorithms for anomaly detection and automated log summarization, reducing manual intervention and alert fatigue by focusing on critical patterns.

    • Continuously monitor and update AI-driven models to adapt to evolving log data trends, ensuring that SIEM ingested data remains actionable and cost-efficient.

By following these best practices, your Syslog parser in Observo AI pipelines will ensure optimized SIEM ingestion, real-time security monitoring, and efficient log management for enhanced cybersecurity and observability.

Related Functions

  • JSON Parser: Parse log events into structured JSON.

  • CEF Parser: Extracts and normalizes fields from CEF-formatted logs, enabling efficient search, correlation, and analysis in SIEM systems.

PreviousPalo Alto ParserNextXML

Last updated

Was this helpful?