Syslog

The Syslog Source is designed to ingest Syslog messages into Observo, enabling seamless integration and processing of log data from network devices, servers, and other sources. This configuration allows you to set up the necessary parameters for the Syslog source, ensuring secure and efficient data ingestion.

Purpose

The purpose of the Observo AI Syslog source is to collect logs from network devices, firewalls, servers, and other systems that transmit data using the standard Syslog protocol. It supports both UDP and TCP transport, ensuring compatibility with a wide range of infrastructure components. This enables centralized ingestion and processing of security and operational logs within Observo’s observability pipeline for enrichment, reduction, and compliance.

Prerequisites

Before configuring the Syslog Source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:

• Observo AI Platform Setup:

  • The Observo AI Site must be installed and available.

  • Verify data supports syslog data formats defined by RFC 3164 and RFC 5424, including optional message-length prefixes (RFC 5425 and RFC 6587) for accurate data transmission.

• Network and Connectivity:

  • Ensure Observo AI can receive syslog data over UDP or TCP. Default ports are UDP 9514, TCP 9514, and TCP+TLS 6514. For devices hardcoded to port 514, enable the built-in in_syslog_default source to forward traffic to port 10514.

  • Check for firewall rules, proxy settings, or VPC configurations that may affect connectivity to the specified ports.

  • If using TLS, ensure certificates are properly configured such as CA certificates for secure communication.

• Authentication (Optional for TLS):

  • For TLS-enabled syslog sources, prepare one of the following authentication methods:

    • Certificate-Based Authentication: Provide paths to CA certificate, host certificate, and private key files if required.

    • No Authentication: If TLS is disabled, no additional credentials are needed.

• Load Balancer (Optional):

  • For high-volume TCP syslog data, configure a load balancer such as HAProxy, nginx, AWS ELB to distribute traffic across Observo AI worker nodes to prevent CPU strain on a single node.

  • If using a load balancer, consider enabling Proxy Protocol v1 or v2 to preserve the original sender IP address.

Prerequisite	Description	Notes
Network	Connectivity to UDP/TCP ports	Default ports: UDP 9514, TCP 9514, TCP+TLS 6514; check firewall/proxy
Authentication	TLS certificate setup if enabled	Provide CA, host cert, and key files for TLS
Load Balancer	Optional for high-volume TCP data	Use HAProxy, nginx, or AWS ELB; enable Proxy Protocol if needed

Integration

The Integration section outlines default configurations for the Observo AI Syslog Source. To configure Syslog as a source in Observo AI, follow these steps to set up and test the data flow:

1. Log in to Observo AI:

   • Navigate to the Sources tab.

   • Click the "Add Source" button and select "Create New".

   • Choose "Syslog" from the list of available sources to begin configuration.

2. General Settings:

   • Name: A unique identifier for the source such as syslog-source-1.

   • Description (Optional): Description for the source

   • Socket address: Socket address to listen for connections on. It should be in the format of host:port. Port should be in range[10000-10200]

     Example
     0.0.0.0:10000

   • The type of socket to use: Select either Listen on a TCP socket or Listen on a UDP socket from the dropdown.

     Select an Option
     Listen on a TCP socket
     Listen on a UDP socket

3. TLS Options (Optional):

   • Determines if TLS is enabled. Relevant when mode is set to tcp (False): Toggle to Enable.

   • Absolute path to a CA certificate file. Relevant when mode is set to tcp. (Empty): Absolute path to an additional CA certificate file

   • Absolute path to a certificate file. Relevant when mode is set to tcp. (Empty): Absolute path to a certificate file to identify the server

   • Absolute path to a private key file. Relevant when mode is set to tcp (Empty): Absolute path to a private key file

   • Private key password. Relevant when mode is set to tcp.: Passphrase to unlock the encrypted key file, if applicable.

   • Verify client certificates. Relevant when mode is set to tcp. If enabled, the certificate must be issued by a trusted issuer. (False): Enable to verify client certificates.

   • Verify client hostname. Relevant when mode is set to tcp. If enabled, the client hostname must match the certificate hostname. (False): Enable to verify client hostname.

4. Advanced Settings:

   • Max number of concurrent TCP connections: 10000 (Default)

   • Time to wait before sending TCP keepalive probes: 600 (Default)

   • The maximum buffer size in bytes of incoming messages: 102400 (Default)

5. Parser Config:

   • Enable Source Log Parser: Disabled by default. Toggle to enable and select a parser from the Source Log Parser dropdown.

   • Add additional parsers as needed for specific syslog formats.

6. Pattern Extractor:

   • See Pattern Extractor for details.

7. Archival Destination:

   • Toggle Enable Archival on Source Switch to enable

   • Under Archival Destination, select from the list of Archival Destinations

8. Save and Test Configuration:

   • Save the configuration settings.

   • Send sample syslog data such as via a syslog sender and verify ingestion in Observo AI by monitoring the Analytics tab in the target pipeline.

Example Scenarios

SecureNet, a fictitious Technology enterprise, wants to integrate their network devices, which send syslog messages over TCP with TLS, into the Observo platform for centralized log analysis and compliance monitoring. They configure the Syslog source to listen on a specific socket address, use TCP as the transport protocol, and enable TLS with certificate-based authentication. The configuration includes verifying client certificates and hostnames to ensure secure communication. The certificates and keys are stored in a secure directory on the Observo server.

Standard Syslog Source Setup

Here is a standard Syslog Source configuration example. Only the required sections and their associated field updates are displayed in the table below:

General Settings

Field	Value	Notes
Name	syslog-secure-net-tcp	Unique identifier for the Syslog source, indicating SecureNet’s TCP-based setup.
Description	Collect TCP syslog logs from SecureNet devices with TLS	Optional, provides context for the source’s purpose.
Socket address	192.168.1.100:10050	Socket address in host:port format, within the allowed range [10000-10200].
The type of socket to use	Listen on a TCP socket	Selected from the dropdown to match the requirement for TCP transport.

Authentication

Field	Value	Notes
Username	syslog_user	Username for authentication, as required for the Syslog source.
Password	S3cur3P@ssw0rd	Password for authentication, ensuring secure access to the Syslog source.

TLS Configuration

Field	Value	Notes
Determines if TLS is enabled	True	Enabled as required, relevant since the mode is set to TCP.
Absolute path to a CA certificate file	/etc/observo/certs/ca-cert.pem	Path to the CA certificate file for verifying client certificates.
Absolute path to a certificate file	/etc/observo/certs/server-cert.pem	Path to the server certificate file to identify the Observo AI server.
Absolute path to a private key file	/etc/observo/certs/server-key.pem	Path to the private key file for the server certificate.
Private key password	KeyP@ss123	Passphrase to unlock the encrypted private key file, if applicable.
Verify client certificates	True	Enabled to ensure the client certificate is issued by a trusted issuer.
Verify client hostname	True	Enabled to ensure the client hostname matches the certificate hostname.

Advanced Settings

Field	Value	Notes
Max number of concurrent TCP connections	10000	Default value for handling concurrent TCP connections.
Time in seconds to wait before sending TCP keepalive probes	600	Default value for TCP keepalive probes.
The size of the receive buffer used for each connection	65536	Default buffer size for each connection.

Test Configuration:

• Click “Save” to store the configuration settings in Observo AI.

• Send sample syslog data using a syslog sender (e.g., via logger or a network device) to 192.168.1.100:10050. Verify ingestion by monitoring the Analytics tab in the Observo pipeline for event counts and throughput.

Notes:

• Authentication: The username and password are fictional but follow secure practices. SecureNet would configure these credentials in their syslog sender and Observo to match.

• TLS Configuration: The certificate and key file paths are examples assuming a secure directory on the Observo AI server. SecureNet must ensure these files are accessible and correctly formatted.

• Network: The socket address (192.168.1.100:10050) is within the allowed port range [10000-10200] and assumes the Observo server is reachable by the syslog senders. Firewall rules must allow TCP traffic on port 10050.

• Troubleshooting: If issues occur (e.g., “No data received” or “TLS errors”), SecureNet should verify the socket address, port, TLS certificates, and network connectivity using tools like nc or telnet, as outlined in the Troubleshooting section.

• Resources: For further details, SecureNet can refer to the Resources section, including RFC 5424 for syslog message formats and RFC 5425 for TLS transport mapping.

Troubleshooting

If issues arise with the Syslog Source in Observo AI, use the following steps to diagnose and resolve them:

• Verify Configuration Settings:

  • Ensure fields like Input ID, Address, UDP/TCP ports, and Octet Count Framing match the syslog sender’s configuration.

  • Confirm that the correct ports such as 9514 for UDP/TCP, 6514 for TLS are open and accessible.

• Check Network Connectivity:

  • Verify that firewall rules, proxy settings, or VPC configurations allow traffic to the specified UDP/TCP ports.

  • Test connectivity using tools like nc (netcat) or telnet to ensure the Observo AI instance can receive data on the configured ports.

• Validate TLS Configuration:

  • For TLS-enabled sources, ensure CA, certificate, and key files are correctly specified and accessible.

  • Check for TLS version mismatches such as sender using TLS 1.2 while Observo AI expects TLS 1.3. Adjust TLS settings or disable "Verify Hostname" if certificate issues occur.

• Monitor Logs and Data:

  • Verify data ingestion by monitoring the Analytics tab in the Observo AI pipeline for throughput and event counts.

  • Check Observo AI logs for errors related to parsing, buffering, or connection issues.

• Common Error Messages:

  • "No data received": Ensure the syslog sender is pointing to the correct IP/port and that network connectivity is not blocked. Verify the sender’s protocol (UDP/TCP) matches the source configuration.

  • "Invalid message format": Confirm that the syslog messages comply with RFC 3164 or RFC 5424. If non-compliant, messages may appear in the _raw field without parsed fields. Use the cribl-syslog-input pack to preview and adjust parsing.

  • "High CPU usage": For high-volume TCP syslog, enable load balancing to distribute traffic across worker nodes.

  • "Events dropped (UDP)": Enable Persistent Queue (PQ) in Always On mode to minimize data loss during backpressure. Set Max buffer size to 0 for UDP sources with PQ enabled.

Issue	Possible Cause	Resolution
No data received	Incorrect IP/port or network block	Verify sender configuration and network connectivity
Invalid message format	Non-RFC-compliant messages	Use cribl-syslog-input pack to adjust parsing
High CPU usage	Single worker overloaded (TCP)	Enable TCP load balancing
Events dropped (UDP)	Backpressure or buffer overflow	Enable Persistent Queue and set Max buffer size to 0
TLS errors	Certificate or version mismatch	Check TLS settings and certificate files

Resources

For additional guidance and detailed information, refer to the following resources:

• External References:

  • What is Syslog? Components, benefits, and best practices

  • RFC 3164 and RFC 5424: Standards for syslog message formats:

    • RFC 3164 (The BSD syslog Protocol)

    • RFC 5424 (The Syslog Protocol)

    • Note: These are authoritative URLs from the Internet Engineering Task Force (IETF), which publishes RFC standards.

  • RFC 5425 and RFC 6587: Specifications for message-length prefixes and transport protocols

    • RFC 5425 (Transport Layer Security (TLS) Transport Mapping for Syslog)

    • RFC 6587 (Transmission of Syslog Messages over TCP)

    • Note: These are authoritative IETF URLs for the respective RFCs.

• Best Practices:

  • Use TCP with TLS for reliable and secure data delivery unless the sender only supports UDP or dropped events are acceptable.

  • Configure separate Syslog Sources for UDP and TCP traffic to enhance security and performance.

  • For high-volume environments, deploy a load balancer to distribute TCP syslog traffic across worker nodes.