Splunk TCP

The Splunk TCP Source in Observo AI enables secure and reliable ingestion of Splunk telemetry data via TCP connections for real-time observability and anomaly detection.

Purpose

The Splunk TCP Source in Observo AI enables the ingestion of Splunk telemetry data, such as system logs and network activity, for observability and anomaly detection. It supports secure TCP connections, often using TLS encryption, to ensure reliable and protected data transmission from Splunk Universal or Heavy Forwarders to the Observo AI platform. This source is critical for organizations that facilitate unified monitoring of large-scale deployments. Proper configuration of network, TLS, and forwarder settings ensures seamless data integration and real-time analytics.

Prerequisites

Before configuring the Splunk TCP Source in Observo AI, ensure the following prerequisites are met:

• Observo AI Platform Setup:

  • The Observo AI Site must be installed and operational.

• Network and Connectivity:

  • Ensure Observo AI is reachable on the TCP port configured for the Splunk TCP input.

    • The default TCP port range is typically [10000-10200].

  • Validate firewall rules and VPC/subnet-level routing to ensure traffic reaches the Observo AI Splunk TCP listener.

• Authentication (Optional for TLS-enabled TCP communication):

  • Ensure TLS certificates (CA, server certs, keys) are properly configured

  • Certificate authentication: CA certificate, host certificate, and private key file.

• Splunk Forwarder Configuration:

  • Configure outputs.conf on the Splunk Universal Forwarder or Heavy Forwarder to forward data to the Observo AI Splunk TCP Source.

  • Specify correct IP and port of the Observo AI Site in the tcpout stanza.

• Load Balancer (Optional):

  • For high-ingest environments, use a TCP-aware load balancer such as HAProxy, AWS NLB to distribute connections across multiple Observo AI worker nodes.

  • If applicable, enable Proxy Protocol to retain original client IP metadata.

Prerequisite	Description	Notes
Network Connectivity	TCP connection to Observo AI source endpoint	Validate port open on firewall or proxy
Authentication (Optional)	TLS certificate configuration for secure TCP	Optional for encrypted connections
Forwarder Configuration	Correct outputs.conf for Splunk TCP forwarding	Match IP/port and configure index/sourcetype
Load Balancer (Optional)	TCP-aware load balancer for traffic distribution	Consider Proxy Protocol to retain client IPs

Integration

To integrate the Splunk TCP Source into Observo AI, follow the steps below:

1. Configure Observo AI TCP Source:

   • Navigate to Sources in the Observo AI UI.

   • Click Add Source > Create New.

   • Choose Splunk TCP as the Source Type.

2. General Settings:

   • Name: Provide a unique name such as splunk-tcp-source-1.

   • Description (Optional): A brief description of the source.

   • Socket Address: Format as host:port. Port must match (UF/HF) forwarder outputs.config.

     Example
     0.0.0.0:10000

3. TLS Options (Optional):

   • TLS Enabled: Enable TLS for encrypted TCP streams.

   • TLS CA File (Empty): Absolute path to an additional CA certificate file. The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

     Example
     /path/to/certificate_authority.crt

   • TLS Crt File (Empty): Absolute path to a certificate file used to identify this server. The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format. If this is set, and is not a PKCS#12 archive, key_file must also be set.

     Example
     /path/to/host_certificate.crt

   • TLS Key File (Empty): Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

     Example
     /path/to/host_certificate.key

   • TLS Verify Certificate (Disabled): Enables certificate verification. If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate. Relevant for both incoming and outgoing connections. If using TLS do NOT disable this unless you understand the risks of not verifying the validity of certificates.

   • TLS Verify Hostname(Disabled): Enables hostname verification. If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension. Only relevant for outgoing connections. If using TLS do NOT disable this unless you understand the risks of not verifying the remote hostname.

4. Forwarder Configuration (Splunk):

   Example	Configuration
   On the Splunk Universal/Heavy Forwarder	# outputs.conf [tcpout] defaultGroup = observo_tcp_group [tcpout:observo_tcp_group] server = <observo_ip>:10000
   If ACK's are needed	# outputs.conf [tcpout] defaultGroup = observo_tcp_group [tcpout:observo_tcp_group] server = <observo_ip>:10000 useACK = true Note: By default, Splunk UF sets useACK as false.

   • Restart the Splunk forwarder after saving changes.

5. Save and Test:

   • Save the source configuration in Observo AI.

   • Send test events from the Splunk forwarder.

   • Confirm data arrival in the Analytics or Pipeline tab.

Example Scenarios

Enerex, a fictitious utility enterprise, integrates Splunk telemetry data via a TCP source into Observo AI for observability and anomaly detection, supporting the deployment of millions of smart devices. The configuration uses a TCP connection on observo.enerex.com:10000 with TLS enabled for secure data transmission, ensuring reliable ingestion of system logs and network activity.

Standard Splunk TCP Source Setup

Here is a standard Splunk TCP Source configuration example. Only the required sections and their associated field updates are displayed in the table below:

General Settings

Field	Value	Notes
Name	splunk-tcp-enerex-1	Unique identifier for the Splunk TCP source, indicating Enerex’s telemetry ingestion.
Description	Ingest Splunk telemetry data from Enerex's smart device logs	Optional, provides context for the source’s purpose.
Socket Address	observo.enerex.com:10000	Host and port for the Observo AI TCP listener, matching the Splunk forwarder’s outputs.conf.

TLS Configuration

Field	Value	Notes
TLS Enabled	True	Enabled to require TLS for secure TCP connections, ensuring encrypted data transfer.
TLS CA File	-----BEGIN CERTIFICATE----- MIID... (PEM format)	Inline PEM string for the CA certificate, verifying the Splunk forwarder’s certificate.
TLS Crt File	-----BEGIN CERTIFICATE----- MIIC... (PEM format)	Inline PEM string for the Observo AI server certificate, identifying the TCP listener.
TLS Key File	-----BEGIN PRIVATE KEY----- MIIE... (PEM format)	Inline PEM string for the private key, securely stored for the server certificate.
TLS Verify Certificate	True	Enables verification of the Splunk forwarder’s certificate, ensuring it’s valid and trusted.
TLS Verify Hostname	True	Ensures the hostname (observo.enerex.com) matches the certificate presented by the forwarder.

Forwarder Configuration (Splunk)

Description	Details
Configure Splunk Forwarder	On the Splunk Universal/Heavy Forwarder, edit outputs.conf: [tcpout] defaultGroup = observo_tcp_group [tcpout:observo_tcp_group] server = observo.enerex.com:10000 useACK = true
Enable TLS	Configure Splunk to use TLS by setting SSL parameters in outputs.conf such as sslCertPath, sslRootCAPath to match Observo AI’s TLS settings.
Restart Forwarder	Restart the Splunk forwarder to apply changes and begin forwarding data to Observo AI.

Test Configuration:

• Click “Save” to store the configuration settings in Observo AI.

• Send test events such as simulated smart device logs from the Splunk forwarder. Verify ingestion by monitoring the Analytics or Pipeline tab in Observo AI for event counts and throughput.

Notes:

• TLS Configuration: PEM certificate/key strings are placeholders; actual values must be provided by Enerex’s security team, securely stored in Observo AI. TLS Enabled, TLS Verify Certificate, and TLS Verify Hostname are set to True for production-grade security.

• Splunk Forwarder: The outputs.conf configuration uses useACK = true to ensure reliable data delivery, critical for high-volume smart device telemetry. TLS settings on the Splunk side must match Observo AI’s certificates.

• Network: Ensure firewall rules allow TCP traffic on port 10000 to observo.enerex.com. For high-ingest scenarios, consider a TCP-aware load balancer such as AWS NLB with Proxy Protocol enabled.

• Troubleshooting: If issues occur such as “No data received” or “TLS handshake failure”, verify the socket address, firewall rules, and TLS certificate paths. Use openssl s_client -connect observo.enerex.com:10000 for TLS debugging and check Observo AI’s Monitoring view for errors, as per the Troubleshooting section.

• Resources: Refer to Splunk Forwarding Data Documentation and Splunk TCP Forwarding Setup for guidance on configuring outputs.conf.

• Enerex Context: This configuration supports ingestion of Splunk telemetry alongside Azure Event Hubs and CrowdStrike data, enabling unified observability for smart device operations and security monitoring.

Troubleshooting

If issues arise with the Splunk TCP Source in Observo AI, consider the following:

• Configuration Checks:

  • Confirm that the port in outputs.conf matches the socket address in the source config.

  • Ensure the Splunk forwarder is running and properly connected.

• Network Validation:

  • Test port accessibility using telnet or nc (netcat).

  • Check firewall rules and security group configurations.

• TLS Debugging (If Enabled):

  • Ensure all certificate files are present and readable.

  • Validate that both server and client support the same TLS version.

  • Use openssl s_client -connect <host>:<port> to diagnose TLS handshake.

• Log Monitoring:

  • Check Observo AI pipeline Monitoring view for parsing errors or dropped connections.

  • Verify ingestion status via Analytics and Data Insights tab.

• Common Errors:

Issue	Possible Cause	Resolution
No data received	Incorrect IP/port or firewall blocking traffic	Confirm outputs.conf and check network access
TLS handshake failure	Certificate mismatch or version incompatibility	Validate certificate paths and TLS versions
Invalid message format	Non-Splunk TCP stream or corrupted payload	Ensure Splunk is sending standard TCP-formatted data

Resources

For additional guidance and detailed information, refer to the following resources:

• External References:

  • Splunk TCP Forwarding Setup (outputs.conf)

  • Splunk Forwarding Data Documentation

• Best Practices:

  • Use TLS for secure TCP connections

  • Configure separate sources for different data types to simplify parsing logic.

  • Monitor connection count and throughput to proactively adjust source settings.