Socket

The Observo AI Socket destination enables real-time forwarding of observability and security events to a custom TCP or UDP network endpoint, with configurable encoding, framing, buffering, and optional TLS for secure, reliable log transmission.

Purpose

The Observo AI Socket destination enables users to send observability and security events to a network endpoint using TCP or UDP protocols for real-time log processing. This integration facilitates flexible data forwarding to custom servers or applications, supporting various encoding and framing options for compatibility. It ensures reliable event delivery with configurable buffering and TLS for secure transmission.

Prerequisites

Before configuring the Socket destination in Observo AI, ensure the following requirements are met to facilitate seamless event data transmission:

  • Socket Endpoint:

    • Set up a server or application listening on a specific IP address or hostname and port (e.g., 92.12.333.224:5000 or https://somehost:5000) to receive log events.

    • Ensure the endpoint is active and configured to accept TCP or UDP connections, depending on the chosen mode.

    • Verify the endpoint supports the intended encoding (Text or JSON) and framing method.

  • Network Access:

    • Confirm that the Observo AI instance can communicate with the Socket endpoint over the specified port (e.g., 5000).

    • Check for firewall rules or network policies that may block outbound TCP or UDP traffic to the endpoint.

  • Encoding and Framing Requirements:

    • Determine the encoding (Text for message field only, JSON for full event payload) and framing method (Raw, Single Character Delimited, Newline Delimited) required by the receiving endpoint.

    • Optionally, prepare an Avro schema if using Avro encoding for structured data.

Prerequisite
Description
Notes

Socket Endpoint

Receives log events via TCP or UDP

Must be active and configured for the chosen protocol

Network Access

Enables communication with the endpoint

Ensure connectivity to the specified IP/hostname and port

Encoding and Framing Requirements

Defines data format and structure

Choose Text or JSON encoding; select appropriate framing method

Integration

The Integration section outlines default configurations for the Socket destination. To tailor the setup to your environment, consult the documentation for your receiving server or application for specific requirements. To configure Socket as a destination in Observo AI, follow these steps:

  1. Log in to Observo AI:

    • Navigate to the Destinations tab.

    • Click the Add Destinations button and select Create New.

    • Choose Socket from the list of available destinations to begin configuration.

  2. General Settings:

    • Name: Add a unique identifier, such as Socket-events-1.

    • Description (Optional): Provide a description, e.g., “Sends observability and security events to Socket.”

    • Mode: The type of socket to use (TCP/UDP).

    • Address: The address to connect to (IP address / hostname) and Port. MUST include a port.

      Example

      92.12.333.224:5000

      https://somehost:5000

    • Keepalive Time Secs: The time to wait before starting to send TCP keepalive probes on an idle connection.

  3. Acknowledgement:

    • Acknowledgements Enabled (False): Whether or not end-to-end acknowledgements are enabled. When enabled, any source connected to this supporting end-to-end acknowledgements, will wait for events to be acknowledged by the sink before acknowledging them at the source.

  4. Encoding:

    • Encoding Codec: The codec to use for encoding events. Default: JSON Encoding.

      Options
      Sub-Options

      JSON Encoding

      Pretty JSON (False): Format JSON with indentation and line breaks for better readability. Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      logfmt Encoding

      Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      Apache Avro Encoding

      Avro Schema: Specify the Apache Avro schema definition for serializing events. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      Newline Delimited JSON Encoding

      Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (Default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      No encoding

      Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      Plain text encoding

      Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      Parquet

      Include Raw Log (False): Capture the complete log message as an additional field(observo_record) apart from the given schema. Examples: In addition to the Parquet schema, there will be a field named "observo_record" in the Parquet file. Parquet Schema: Enter parquet schema for encoding. Examples: message root { optional binary stream; optional binary time; optional group kubernetes { optional binary pod_name; optional binary pod_id; optional binary docker_id; optional binary container_hash; optional binary container_image; optional group labels { optional binary pod-template-hash; } } } Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      Common Event Format (CEF)

      CEF Device Event Class ID: Provide a unique identifier for categorizing the type of event (maximum 1023 characters). Example: login-failure CEF Device Product: Specify the product name that generated the event (maximum 63 characters). Example: Log Analyzer CEF Device Vendor: Specify the vendor name that produced the event (maximum 63 characters). Example: Observo CEF Device Version: Specify the version of the product that generated the event (maximum 31 characters). Example: 1.0.0 CEF Extensions (Add): Define custom key-value pairs for additional event data fields in CEF format. CEF Name: Provide a human-readable description of the event (maximum 512 characters). Example: cef.name CEF Severity: Indicate the importance of the event with a value from 0 (lowest) to 10 (highest). Example: 5 CEF Version (Select): Specify which version of the CEF specification to use for formatting. - CEF specification version 0.1 - CEF specification version 1.x Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      CSV Format

      CSV Fields (Add): Specify the field names to include as columns in the CSV output and their order. Examples: - timestamp - host - message CSV Buffer Capacity (Optional): Set the internal buffer size (in bytes) used when writing CSV data. Example: 8192 CSV Delimitier (Optional): Set the character that separates fields in the CSV output. Example: , Enable Double Quote Escapes (True): When enabled, quotes in field data are escaped by doubling them. When disabled, an escape character is used instead. CSV Escape Character (Optional): Set the character used to escape quotes when double_quote is disabled. Example: <br> CSV Quote Character (Optional): Set the character used for quoting fields in the CSV output. Example: " CSV Quoting Style (Optional): Control when field values should be wrapped in quote characters. Options: - Always quot all fields - Quote only when necessary - Never use quotes - Quote all non-numeric fields Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      Protocol Buffers

      Protobuf Message Type: Specify the fully qualified message type name for Protobuf serialization. Example: package.Message Protobuf Descriptor File: Specify the path to the compiled protobuf descriptor file (.desc). Example: /path/to/descriptor.desc Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

      Graylog Extended Log Format (GELF)

      Encoding Avro Schema (Optional): The Avro schema. Example: { "type": "record", "name": "log", "fields": [{ "name": "message", "type": "string" }] } Encoding Metric Tag Values (Select): Controls how metric tag values are encoded. - Tag values will be exposed as single strings (default) - Tags exposed as arrays of strings Note: When set to single, only the last non-bare value of tags will be displayed with the metric. When set to full, all metric tags will be exposed as separate assignments. Encoding Timestamp Format (Select): - RFC3339 format - UNIX format

  5. Framing:

    • Framing Character Delimited Delimiter: The ASCII (7-bit) character that delimits byte sequences.

    • Framing Method: Select the framing method (Raw Event data, Single Character Delimited, Newline Delimited).

      • Default: Newline Delimited.

  6. TLS Configuration (Optional):

    • TLS CA: Provide the CA certificate as an inline string in PEM format, if using a custom certificate authority.

    • TLS CRT: Provide the certificate as a string in PEM format, if applicable.

    • TLS Key: Provide the key as a string in PEM format, if applicable.

    • TLS Verify Certificate: Enable certificate verification (True/False).

      • Default: False (set to True for secure connections).

    • TLS Verify Hostname: Enable hostname verification (True/False).

      • Default: False (set to True for secure connections).

  7. Buffering:

    • Buffer Type: Specifies the buffering mechanism for event delivery. Default: Empty

      Options
      Description

      Memory

      High-Performance, in-memory buffering Max Events: The maximum number of events allowed in the buffer. Default: 500 When Full: Event handling behavior when a buffer is full. Default: Block - Block: Wait for free space in the buffer.This applies backpressure up the topology, signalling that sources should slow down the acceptance/consumption of events. This means that while no data is lost, data will pile up at the edge. - Drop Newest: Drop the event instead of waiting for free space in the buffer. The event will be intentionally dropped. This mode is typically used when performance is the highest priority, and it is preferable to temporarily lose events rather than cause a slowdown in the acceptance/consumption of events.

      Disk

      Lower-Performance, Less-costly, on disk buffering Max Bytes Size: The maximum number of bytes size allowed in the buffer. Must be at-least 268435488 When Full: Event handling behavior when a buffer is full. Default: Block - Block: Wait for free space in the buffer. This applies backpressure up the topology, signalling that sources should slow down the acceptance/consumption of events. This means that while no data is lost, data will pile up at the edge. - Drop Newest: Drop the event instead of waiting for free space in the buffer. The event will be intentionally dropped. This mode is typically used when performance is the highest priority, and it is preferable to temporarily lose events rather than cause a slowdown in the acceptance/consumption of events.

  8. Advanced Settings:

    • Send Buffer Bytes: The size of the socket's send buffer. - If set, the value of the setting is passed via the SO_SNDBUF option.

      Example

      65536

  9. Save and Test Configuration:

    • Save the configuration settings in Observo AI.

    • Send sample log data and verify it appears in the Socket platform under the specified tags or event type.

Example Scenario

To illustrate the Socket destination’s functionality, consider a scenario where you configure Observo AI to send security logs to a custom TCP server for real-time analysis:

In Observo AI, create a pipeline with a Source for collecting security logs from a network service. Then add a Socket destination with the following settings, using your specific Socket values:

General Settings

Field
Value
Description

Name

Socket-security-events

Unique identifier for the destination.

Description

Send security events logs to Socket.

Provides context for the destination's purpose.

Endpoint

92.12.333.224:5000

Specify the address and port to connect to

Mode

TCP

Define the network protocol for connecting.

Encoding

Field
Value
Description

Encoding Codec

json

Select json for the codec.

Encoding Timestamp Format

RFC 3339

Select the format used for timestamp fields

Framing Method

Field
Value
Description

Framing Method

Newline Delimited

Select the framing method

Test Configuration:

  • Save settings, route the pipeline’s output to the Socket destination

  • Send sample security log data (e.g., firewall events) through the pipeline.

  • At the receiving TCP server, verify that the logs arrive in JSON format with RFC 3339 timestamps and are newline-delimited, ensuring proper parsing and storage.

  • This setup enables real-time log forwarding with secure transmission to a custom server.

Troubleshooting

If issues arise with the Socket destination, use the following steps to diagnose and resolve them:

  • Verify Configuration Settings:

    • Ensure the Mode (TCP/UDP) and Address (IP/hostname and port) are correctly configured and match the receiving server’s setup.

    • Confirm that the Encoding Codec and Framing Method align with the server’s requirements.

  • Check Network Connectivity:

    • Verify that the Observo AI instance can reach the Socket endpoint (e.g., 92.12.333.224:5000) over the specified protocol and port.

    • Use tools like telnet or nc to test connectivity and check for firewall rules or network policies blocking traffic.

  • Monitor Logs:

    • Check Observo AI logs for errors or warnings related to event transmission to the Socket endpoint.

    • On the receiving server, inspect logs or use a packet capture tool to confirm that events are arriving with the expected format.

  • Validate Network Connectivity:

    • Ensure the Encoding Codec (Text/JSON) and Framing Method (Raw/Single Character/Newline Delimited) are compatible with the server’s parser.

    • If using Avro, verify the schema matches the event structure.

  • Test Data Flow:

    • Send sample log data through Observo AI and monitor its arrival at the Socket endpoint.

    • Use the Analytics tab in the targeted Observo AI pipeline to monitor data volume and ensure expected throughput.

  • Check Buffer and Backpressure:

    • If events are delayed or dropped, verify the Buffer Type and When Full settings (Block or Drop Newest).

    • Adjust Max Events or Max Bytes Size to resolve backpressure issues.

Issue
Possible Cause
Resolution

Events not appearing at endpoint

Incorrect Mode or Address

Verify Mode (TCP/UDP) and Address configuration

Connection failures

Network or firewall issues

Test connectivity with telnet/nc; check network policies

Incorrect event format

Mismatched Encoding Codec or Framing

Align codec and framing with server requirements

Slow event transfer

Buffer full or backpressure

Adjust Buffer Type, Max Events, or When Full settings

Resources

For additional guidance and detailed information, refer to the following resources:

PreviousNew RelicNextTransforms

Last updated

Was this helpful?