Logstash

The Observo AI Logstash source enables real-time ingestion of logs, events, and metrics from a Logstash instance into the Observo AI platform, supporting formats like JSON, CSV, or plain text via HTTP or TCP output plugins for enhanced observability, monitoring, and analytics.

Purpose

The purpose of the Observo AI Source Logstash is to enable users to ingest logs, events, and metrics from a Logstash instance into the Observo AI platform for analysis and processing. It facilitates the collection of data in formats such as JSON, CSV, or plain text, sent via Logstash output plugins (e.g., HTTP or TCP), allowing organizations to streamline data pipelines, enhance observability, and support use cases such as monitoring, analytics, and troubleshooting by processing Logstash data in real time.

Prerequisites

Before configuring the Logstash source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:

  • Observo AI Platform Setup:

    • The Observo AI platform must be installed and operational, with support for the Logstash source.

    • Verify that the platform can process data in formats commonly sent by Logstash, such as JSON, CSV, or plain text.

  • Logstash Configuration:

    • A running Logstash instance must be configured to send logs, events, or metrics to Observo AI via an output plugin such as HTTP or TCP output.

    • Obtain the endpoint URL for Observo AI such as https://your-observo-instance:9600/logstash where data will be sent.

  • Authentication:

    • Prepare one of the following authentication methods:

      • Basic Authentication: Provide a username and password for HTTP Basic Auth, if required.

      • No Authentication: If no authentication is configured, ensure the Logstash client can send data without credentials.

  • Network and Connectivity:

    • Ensure the Logstash instance can communicate with the Observo AI endpoint over HTTP/HTTPS or TCP.

    • Check for firewall rules, proxy settings, or VPC configurations that may block traffic to the configured port (default: TCP 9600 for HTTP, 443 for HTTPS).

Prerequisite
Description
Notes

Observo AI Platform

Must be installed and support Logstash source

Verify support for JSON, CSV, or plain text formats

Logstash Config

Running Logstash instance for data submission

Configure output plugin to send to Observo AI endpoint

Authentication

Basic Auth or no authentication

Prepare username/password if required; confirm client config

Network

Connectivity to Observo AI endpoint

Check firewalls, proxies, and VPC for port access (default: 9600)

Logstash Side Configuration Changes

Configuring Logstash to route event data to Observo.

Without TLS

output {
    lumberjack {
        hosts => ["<observo_site_loadbalancer_ip>"]
        port => <observo_site_logstash_source_port>
    }
}

With TLS Copy the CA certificate from Observo UI. Copy the contents of Ca certificate to your Logstash at /etc/ssl/observo.crt

output {
    lumberjack {
        hosts => ["<observo_site_loadbalancer_ip>"]
        port => <observo_site_logstash_source_port>
        ssl_certificate => "/etc/ssl/observo.crt"
    }

Integration

The Integration section outlines the configurations for the Logstash source. To configure the Logstash source in Observo AI, follow these steps to set up and test the data flow:

  1. Log in to Observo AI:

    • Navigate to the Sources tab.

    • Click the Add Source button and select Create New.

    • Choose Logstash from the list of available sources to begin configuration.

  2. General Settings:

    • Name: A unique identifier for the source, such as logstash-source-1.

    • Description (Optional): Provide a description for the source.

    • Socket Address: Socket address to listen for connections on. It should be in the format of host:port. Port should be in range[10000-10200]

      Example

      0.0.0.0:10000

  3. TLS Configuration (Optional):

    • TLS Ca File: Absolute path to an additional CA certificate file. The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format. Default: /etc/certs/ca.crt

      Example

      /path/to/certificate_authority.crt

    • TLS Enable (False): Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

    • TLS Crt File: Absolute path to a certificate file used to identify this server. The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format. If this is set, and is not a PKCS#12 archive, key_file must also be set. Default: /etc/certs/tls.crt

      Example

      /path/to/host_certificate.crt

    • TLS Key File: Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format. Default: /etc/certs/tls.key

      Example

      /path/to/host_certificate.key

    • TLS Verify Certificate (False): Enables certificate verification. If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate. Relevant for both incoming and outgoing connections. Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

    • TLS Verify Hostname (True): Enables hostname verification. If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension. Only relevant for outgoing connections. Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

  4. Advanced Settings (Optional):

    • Max number of concurrent TCP connections: Default: 10000

    • Time in seconds to wait before sending TCP keepalive probes: Default: 600

    • The size of the receive buffer used for each connection: Default: 65536

  5. Parser Config:

    • Enable Source Log Parser: (False)

    • Toggle Enable Source Log Parser Switch to enable.

    • Select appropriate Parser from the Source Log Parser dropdown.

    • Add additional Parsers as needed.

  6. Pattern Extractor:

    • Refer to Observo AI's Pattern Extractor documentation for details on configuring pattern-based data extraction.

  7. Archival Destination:

    • Toggle Enable Archival on Source Switch to enable.

    • Under Archival Destination, select from the list of Archival Destinations (Required).

  8. Save and Test Configuration:

    • Save the configuration settings in Observo AI.

    • Configure your Logstash instance to send data to the Observo AI endpoint such as via the HTTP or TCP output plugin in logstash.conf.

    • Verify ingestion in the Analytics tab for data flow.

Example Scenarios

VitalCare Health Systems, a a fictitious enterprise in the healthcare sector, operates a network of hospitals and clinics. To enhance their observability and ensure compliance with healthcare regulations, VitalCare aims to ingest patient system logs, application metrics, and audit events from their Logstash instance into the Observo AI platform. This integration will enable real-time monitoring, analytics, and troubleshooting of their healthcare IT infrastructure. Below is the detailed configuration process for setting up the Logstash source in Observo AI, based on the provided documentation, with all required fields specified.

Standard Logstash Source Setup

Here is a standard Logstash Source configuration example. Only the required sections and their associated field updates are displayed in the table below:

General Settings

Field
Value
Description

Name

logstash-health-logs

Unique identifier for the Logstash source.

Description

Source for ingesting healthcare system logs and metrics

Optional description for clarity.

Socket Address

0.0.0.0:10020

Listens for Logstash connections on port 10020, within the supported range (10000-10200).

Authentication Settings

Field
Value
Description

Username

vitalcare_admin

Username for HTTP Basic Authentication.

Password

${LOGSTASH_AUTH_PASSWORD}

Password stored securely in Observo AI’s secure storage.

TLS Configuration

Field
Value
Description

TLS Ca File

-----BEGIN CERTIFICATE-----\nMIID...==\n-----END CERTIFICATE-----

Inline CA certificate in PEM format for Logstash connection.

TLS Enable

True

Enables TLS for secure communication.

TLS Crt File

/certs/vitalcare_server_cert.pem

Path to the server certificate in PEM format.

TLS Key File

/certs/vitalcare_server_key.pem

Path to the server private key in PEM format.

TLS Verify Certificate

True

Enables certificate verification for secure connections.

TLS Verify Hostname

True

Verifies the hostname in the client’s TLS certificate.

Advanced Settings

Field
Value
Description

Max number of concurrent TCP connections

5000

Limits to 5000 concurrent connections to manage load.

Time in seconds to wait before sending TCP keepalive probes

300

Sends TCP keepalive probes every 300 seconds to maintain connections.

The size of the receive buffer used for each connection

131072

Receive buffer size of 128 KB for efficient data handling.

Logstash Side Configuration

To route event data to Observo AI, configure the Logstash instance with the following output plugin in logstash.conf:

output {
 lumberjack {
 hosts \=\> \["observo.vitalcarehealth.com"\]
 port \=\> 10020
 ssl_certificate \=\> "/etc/ssl/observo.crt"
 }
}

Ensure the CA certificate from Observo AI is copied to /etc/ssl/observo.crt on the Logstash server.

Test Configuration

  • Configure the Logstash instance to send data to https://observo.vitalcarehealth.com:10020 using the lumberjack output plugin with the specified TLS certificate.

  • Send sample log data (e.g., patient system logs or audit events) and verify ingestion in the Observo AI Analytics tab, checking for expected throughput.

  • Monitor Observo AI logs for errors and confirm data flow from the Logstash instance.

Scenario Troubleshooting

  • Authentication Errors: Ensure vitalcare_admin and ${LOGSTASH_AUTH_PASSWORD} match the credentials in the Logstash output plugin configuration.

  • Connectivity Issues: Verify that port 10020 is open and accessible through firewall rules and proxies. Test with netcat or telnet to observo.vitalcarehealth.com:10020.

  • Data Not Ingested: Confirm that Logstash is sending data in a supported format (e.g., JSON) and that the appropriate parser is enabled in Parser Config.

  • Connection Refused: Ensure Observo AI is listening on 0.0.0.0:10020 and that network configurations allow incoming traffic.

  • TLS Errors: Verify that the CA certificate, server certificate, and key files are correctly specified and that Logstash uses the matching CA certificate.

This configuration enables VitalCare Health Systems to securely ingest Logstash data into Observo AI, supporting real-time monitoring, compliance, and troubleshooting of their healthcare IT infrastructure.

Troubleshooting

If issues arise with the Logstash source in Observo AI, use the following steps to diagnose and resolve them:

  • Verify Configuration Settings:

    • Ensure fields like Address, Port, and Authentication settings match the Logstash output plugin configuration.

    • Confirm the port (default: 9600) is open and accessible.

  • Check Authentication:

    • For Basic Auth, verify the username and password are correct and match the credentials configured in the Logstash output plugin.

    • Check Observo AI logs for authentication failure errors.

  • Validate Network Connectivity:

    • Ensure firewall rules, proxy settings, or VPC configurations allow traffic from the Logstash instance to the Observo AI endpoint.

    • Test connectivity using tools like curl, netcat, or telnet to the configured address and port.

  • Common Error Messages:

    • "Connection refused": Indicates the port is not open or Observo AI is not listening. Verify Address, Port, and firewall settings.

    • "Authentication failed": Confirm the username and password match the Logstash output plugin configuration.

    • "Request too large": Check the Max Body Size setting; increase if incoming data exceeds the limit.

  • Monitor Logs and Data:

    • Verify data ingestion by monitoring the Analytics tab in the targeted Observo AI pipeline for data throughput.

    • Check Observo AI logs for errors or warnings related to data ingestion from the Logstash source.

Issue
Possible Cause
Resolution

Data not ingested

Incorrect address or port config

Verify Address and Port settings

Authentication errors

Invalid or misconfigured credentials

Check username/password and Logstash config

Connectivity issues

Firewall or proxy blocking access

Test network connectivity and check firewall rules

"Connection refused"

Port not open or service not listening

Ensure Observo AI listens on correct address/port

"Authentication failed"

Mismatched credentials

Verify auth settings match Logstash output plugin

"Request too large"

Payload exceeds size limit

Increase Max Body Size in Advanced Settings

Resources

For additional guidance and detailed information, refer to the following resources:

PreviousKafkaNextMicrosoft Entra Log Collector

Last updated

Was this helpful?