Websocket

The WebSocket Source enables the ingestion of real-time streaming log data from WebSocket servers, facilitating continuous data collection for enhanced monitoring, analytics, and observability across various applications and services.

Purpose

The purpose of the WebSocket source is to enable users to ingest streaming log data from WebSocket servers into the Observo AI platform for real-time analysis and processing. It facilitates the collection of structured log events through secure WebSocket connections, allowing organizations to streamline data pipelines, enhance observability, and support use cases such as application monitoring, system analytics, security event tracking, and real-time alerting by processing WebSocket-based data streams from diverse sources.

Prerequisites

Before configuring the WebSocket source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:

• WebSocket Server Configuration:

  • Identify the WebSocket URI endpoint (e.g., wss://logs.example.com/stream) where the source will connect.

  • Ensure the WebSocket server is configured to accept connections from Observo AI.

  • Verify the WebSocket server supports the required protocol (WSS for secure connections).

• Authentication:

  • Prepare authentication credentials as required by the WebSocket server:

    • Bearer Token: Obtain a JWT (JSON Web Token) or OAuth2 bearer token from the service provider.

    • Basic Authentication: Username and password credentials, if applicable.

• Network and Connectivity:

  • Ensure Observo AI can establish outbound connections to the WebSocket server endpoint.

  • Check for firewall rules, proxy settings, or network policies that may affect connectivity.

  • Verify DNS resolution for the WebSocket server hostname.

Prerequisite	Description	Notes
WebSocket URI	Server endpoint for establishing connections	Must include protocol (wss:// or ws://) and full path
Authentication Credentials	Bearer token or basic authentication credentials	Obtain from your WebSocket server provider
Network Connectivity	Outbound access to WebSocket server	Check firewall rules, DNS resolution, and network policies

Integration

The Integration section outlines the configurations for the WebSocket source. To configure the WebSocket source in Observo AI, follow these steps to set up and test the data flow:

1. Log in to Observo AI:

   • Navigate to the Sources tab.

   • Click the Add Source button and select Create New.

   • Choose WebSocket from the list of available sources to begin configuration.

2. General Settings:

   • Name: A unique identifier for the source, such as websocket-app-logs-1.

   • Description (Optional): Provide a description for the source.

   • WebSocket URI: The WebSocket server endpoint to connect to. Must include protocol and host.

     | Example | |-----------------------------------------| | wss://logs.example.com/socket | | ws://127.0.0.1:8080/events | | ws://internal-logs.company.local:8080 |

3. Authentication:

   • Authentication Strategy: Select the authentication method required by the WebSocket server.

     Options
     Basic Authentication
     Bearer Token Authentication

   • Username: Username for basic authentication. Only used when strategy is set to basic.

     Example
     admin_user

   • Password: Password for basic authentication. Only used when strategy is set to basic. (Masked field)

     Example
     SecurePass123

   • Bearer Token: Authentication token (OAuth2, JWT, etc.). Only used when strategy is set to bearer. (Masked field)

     Example
     eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJz...

4. Connection Settings (Optional):

   • Initial Message: Optional message to send upon connection, such as subscription commands.

     Example
     {"action":"subscribe","channel":"events"}
     SUBSCRIBE logs

   • Initial Message Timeout (seconds): Timeout while waiting for reply to initial message. Default: 2

   • Connection Timeout (seconds): Timeout while establishing the WebSocket connection. Default: 30

     Example
     30

   • Ping Interval (seconds): Interval between sending keepalive pings. If not set, pings are not sent automatically.

     Example
     30
     60

   • Ping Timeout (seconds): Time to wait for pong response before reconnecting. Only used if Ping Interval is set.

     Example
     10
     15

   • Custom Ping Message: Application-level ping message. If not set, standard WebSocket ping frames are used.

     Example
     {"type":"ping"}
     PING

5. Decoding:

   • Decoding Codec: The codec for decoding incoming events. Default: Raw Bytes

     Options
     Raw Bytes
     JSON
     GELF
     Syslog (RFC 3164 or RFC 5424)
     Native JSON

   • Character Delimiter: The delimiter character when using character-delimited framing.

     Example
     |
     ,

   • Framing Method: The method for framing messages. Default: Raw Bytes (not delimited)

     Options
     Raw Bytes (not delimited)
     Newline Delimited
     Character Delimited
     Length Delimited (32-bit prefix)
     Octet Counting

6. TLS Configuration (Required for WSS):

   • Enable TLS: Toggle to Enable. Required for wss:// protocol.

   • CA Certificate File: Absolute path to an additional CA certificate file in DER or PEM (X.509) format.

   • Client Certificate File: Absolute path to a client certificate file for mutual TLS authentication in DER, PEM (X.509), or PKCS#12 format.

   • Client Private Key File: Absolute path to the private key file in DER or PEM (PKCS#8) format.

   • Private Key Password: Passphrase to unlock the encrypted key file, if applicable. (Masked field)

   • Verify Server Certificate: Toggle to enable verification of the server's certificate. Recommended for production.

   • Verify Server Hostname: Toggle to enable verification that the server hostname matches the certificate. Recommended for production.

   • Server Name (SNI): Server name for Server Name Indication, typically auto-detected from URI.

7. Parser Config:

   • Enable Source Log Parser: Toggle to enable parser configuration.

   • Recommended: Enable the appropriate parser (e.g. JSON Parser) based on the format of logs from your WebSocket server.

   • Source Log Parser: Select parser from dropdown (e.g. JSON Parser, Syslog Parser).

   • Add additional Parsers as needed using the Add button.

8. Pattern Extractor:

   • Refer to Observo AI's Pattern Extractor documentation for details on configuring pattern-based data extraction.

9. Archival Destination:

   • Toggle Enable Archival on Source to enable.

10. Save and Test Configuration:

    • Click Save to store the configuration settings in Observo AI.

    • Monitor the source connection status and verify data ingestion in the Analytics tab for event counts and throughput.

Troubleshooting

If issues arise with the WebSocket source in Observo AI, use the following steps to diagnose and resolve them:

• Verify Configuration Settings:

  • Ensure the WebSocket URI is correctly entered and includes the proper protocol (wss:// for secure connections, ws:// for non-secure).

  • Confirm authentication credentials (bearer token or username/password) match the expected format and are valid.

  • Verify TLS settings are enabled when using wss:// protocol.

• Check Authentication:

  • Verify the bearer token is valid and not expired. JWT tokens have expiration times that must be monitored.

  • For basic authentication, ensure username and password are correct.

  • Check Observo AI logs for authentication failure messages.

  • Verify system time is synchronized, as JWT validation depends on accurate timestamps.

• Validate Network Connectivity:

  • Test connectivity to the WebSocket server endpoint using tools like curl, websocat, or browser developer tools.

  • Check for firewall rules or network policies that may block outbound WebSocket connections.

  • Verify DNS resolution for the WebSocket server hostname.

  • Ensure proxy settings, if applicable, are configured correctly.

• TLS/SSL Issues:

  • Confirm TLS is enabled in the configuration when using wss:// protocol.

  • Verify certificate validation settings are appropriate for your environment.

  • Check that the server's certificate is valid and not expired.

  • For mutual TLS, ensure client certificates and private keys are correctly configured.

• Common Error Messages:

  • "Connection refused": The WebSocket server is not reachable or not listening. Verify the URI and check network connectivity.

  • "Authentication failed": The bearer token or credentials are invalid or expired. Verify authentication settings and token validity.

  • "TLS handshake failed": TLS configuration issue. Verify TLS is enabled and certificates are valid.

  • "Connection timeout": The connection attempt exceeded the configured timeout. Check network latency and adjust timeout settings if needed.

  • "Ping timeout": Server didn't respond to keepalive pings. Connection will automatically reconnect.

  • "Protocol error": WebSocket protocol violation. Check server compatibility and message formats.

• Monitor Logs and Data:

  • Monitor the Analytics tab in the targeted Observo AI pipeline for data volume and throughput.

  • Review Observo AI logs for errors, warnings, or reconnection attempts related to the WebSocket source.

  • Track authentication success/failure rates and reconnection frequency.

Issue	Possible Cause	Resolution
Data not ingested	Connection not established	Verify WebSocket URI and network connectivity
Authentication errors	Invalid or expired bearer token	Check token validity and authentication strategy settings
Connection failures	Network or firewall blocking access	Test connectivity and check firewall/proxy rules
"Connection refused"	Server unreachable or incorrect URI	Verify URI format and server availability
"Authentication failed"	Invalid credentials or expired token	Verify bearer token or username/password credentials
"TLS handshake failed"	TLS not enabled or certificate issue	Enable TLS and verify certificate configuration
"Connection timeout"	Network latency or incorrect timeout	Check network conditions and adjust timeout settings
Frequent reconnections	Unstable connection or ping issues	Review ping interval/timeout settings and network stability
Message parsing errors	Incorrect codec selected	Verify decoding codec matches data format (e.g., JSON)
Protocol errors	WebSocket protocol mismatch	Verify server supports WebSocket protocol version

Resources

For additional guidance and detailed information, refer to the following resources:

• External Documentation:

  • WebSocket Protocol (RFC 6455)

  • WebSocket API - MDN Web Docs

  • What is WebSocket?

  • Understanding WebSocket Connections

  • What is TLS (Transport Layer Security)?

  • JSON Web Tokens (JWT) Introduction

  • WebSocket Security Best Practices

  • WebSocket Testing Tools