Fluent
This source enables you to ingest data from a FluentD/FluentBit source into Observo. FluentD/FluentBit are popular log aggregators and forwarding agents, and this integration enables you to ingest logs into the Observo platform for centralized observability and monitoring.
Purpose
The purpose of the Observo AI Fluent source is to enable users to ingest logs, events, and metrics from Fluent Bit or Fluentd instances into the Observo AI platform for analysis and processing. It facilitates the collection of data in formats such as JSON, sent via Fluent output plugins (e.g., HTTP or Splunk HEC), allowing organizations to streamline data pipelines, enhance observability, and support use cases such as monitoring, analytics, and troubleshooting by processing Fluent data in real time.
Prerequisites
Before configuring the Fluent source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:
Observo AI Platform Setup:
The Observo AI platform must be installed and operational, with support for the Fluent source.
Verify that the platform can process data in formats commonly sent by Fluent Bit and Fluentd, such as JSON.
Fluent Configuration:
A running Fluent Bit or Fluentd instance must be configured to send logs, events, or metrics to Observo AI via an output plugin, such as the HTTP or Splunk HEC output.
Obtain the endpoint URL for Observo AI, such as https://your-observo-instance:<port>/fluent, where data will be sent.
Authentication:
Prepare one of the following authentication methods:
Basic Authentication: Provide a username and password for HTTP Basic Auth, if required.
No Authentication: If no authentication is configured, ensure the Fluent client can send data without credentials.
Network and Connectivity:
Ensure the Fluent Bit or Fluentd instance can communicate with the Observo AI endpoint over HTTP/HTTPS or TCP.
Check for firewall rules, proxy settings, or VPC configurations that may block traffic to the configured ports
The primary Fleuntd default ports are 24224 (Forward) for the forward protocol and 9880 (HTTP) for HTTP access. The default port for Fluentd's forward protocol, which Fluent Bit uses to send data, is 24224. Fluent Bit also uses port 5170 by default for TCP connections. However, the specific port used depends on the configuration file. See TCP | Fluent Bit: Official Manual. The default port for Fluent Bit when using HTTPS is 8071. For HTTP, the default port is 8070.
Observo AI Platform
Must be installed and support Fluent source
Verify support for JSON format
Fluent Config
Running Fluent Bit or Fluentd instance for data submission
Configure output plugin to send to Observo AI endpoint
Authentication
Basic Auth or no authentication
Prepare username/password if required; confirm client config
Network
Connectivity to Observo AI endpoint
Check firewalls, proxies, and VPC for port access
Integration
The Integration section outlines the configurations for the Fluent source. To configure the Fluent source in Observo AI, follow these steps to set up and test the data flow:
Log in to Observo AI:
Navigate to the Sources tab.
Click the Add Source button and select Create New.
Choose Fluent from the list of available sources to begin configuration.
General Settings:
Name: A unique identifier for the source, such as fluent-source-1.
Description (Optional): Provide a description for the source.
Socket Address: Socket address to listen for connections on. It should be in the format of host:port. The port should be in range [10000-10200].
Example0.0.0.0:10000
Advanced Settings (Optional):
Max number of concurrent TCP connections: Default: 10000
Time in seconds to wait before sending TCP keepalive probes: Default: 1
TLS Configuration (Optional):
TLS Ca File: Absolute path to an additional CA certificate file. The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format. Default: /etc/certs/ca.crt
Example/path/to/certificate_authority.crt
TLS Enable (False): Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
TLS Crt File: Absolute path to a certificate file used to identify this server. The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format. If this is set, and is not a PKCS#12 archive, key_file must also be set. Default: /etc/certs/tls.crt
Example/path/to/host_certificate.crt
TLS Key File: Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format. Default: /etc/certs/tls.key
Example/path/to/host_certificate.key
TLS Verify Certificate (False): Enables certificate verification. If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate. Relevant for both incoming and outgoing connections. Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
TLS Verify Hostname (True): Enables hostname verification. If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension. Only relevant for outgoing connections. Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Parser Config:
Enable Source Log Parser: (False)
Toggle Enable Source Log Parser Switch to enable.
Select appropriate Parser from the Source Log Parser dropdown.
Add additional Parsers as needed.
Pattern Extractor:
Refer to Observo AI's Pattern Extractor documentation for details on configuring pattern-based data extraction.
Archival Destination:
Toggle Enable Archival on Source Switch to enable.
Under Archival Destination, select from the list of Archival Destinations (Required).
Save and Test Configuration:
Save the configuration settings in Observo AI.
Configure your Fluent Bit or Fluentd instance to send data to the Observo AI endpoint, such as via the HTTP or Splunk HEC output plugin in td-agent-bit.conf or fluent.conf.
Verify ingestion in the Analytics tab for data flow.
Example Scenarios
PrecisionForge Industries, a fictitious manufacturing enterprise, specializes in automated production lines for automotive parts. To enhance system monitoring and troubleshoot equipment performance, PrecisionForge integrates the Observo AI platform to ingest logs from a Fluent Bit instance deployed across their factory IoT devices. These logs, in JSON format, capture machine performance metrics and operational events, enabling real-time analytics and predictive maintenance. The IT team configures the Fluent source to receive data via the HTTP output plugin from Fluent Bit, ensuring secure and reliable data ingestion for centralized observability and compliance with industry standards.
Standard Fluent Source Setup
Here is a standard Fluent Source configuration example. Only the required sections and their associated field updates are displayed in the table below:
General Settings
Name
fluent-precisionforge-logs-1
Unique identifier for the Fluent source, specific to PrecisionForge’s log collection.
Description
Machine performance logs from Fluent Bit for IoT devices
Optional description to clarify the purpose of the source.
Socket Address
0.0.0.0:10000
Socket address to listen for HTTP connections from Fluent Bit, using port 10000 within the specified range [10000-10200].
Advanced Settings
Max number of concurrent TCP connections
5000
Reduced from default 10000 to optimize for expected IoT device load.
Time in seconds to wait before sending TCP keepalive probes
2
Set to 2 seconds to ensure timely keepalive probes for stable connections.
TLS Configuration
TLS Ca File
/etc/certs/precisionforge_ca.crt
Path to the CA certificate in PEM format for verifying Fluent Bit client certificates.
TLS Enable
True
Enables TLS for secure incoming connections, critical for manufacturing data security.
TLS Crt File
/etc/certs/precisionforge_tls.crt
Path to the server certificate in PEM format to identify the Observo AI server.
TLS Key File
/etc/certs/precisionforge_tls.key
Path to the private key in PEM format for secure connections.
TLS Verify Certificate
True
Enables certificate verification to ensure Fluent Bit’s certificate is valid and trusted.
TLS Verify Hostname
True
Ensures the hostname in the client certificate matches the Fluent Bit instance.
Troubleshooting
If issues arise with the Fluent source in Observo AI, use the following steps to diagnose and resolve them:
Verify Configuration Settings:
Ensure fields like Socket Address and Authentication settings match the Fluent Bit or Fluentd output plugin configuration.
Confirm the default Fleuntd or Flent Bit ports are open and accessible.
Check Authentication:
For Basic Auth, verify the username and password are correct and match the credentials configured in the Fluent output plugin.
Check Observo AI logs for authentication failure errors.
Validate Network Connectivity:
Ensure firewall rules, proxy settings, or VPC configurations allow traffic from the Fluent Bit or Fluentd instance to the Observo AI endpoint.
Test connectivity using tools like curl, netcat, or telnet to the configured address and port.
Common Error Messages:
"Connection refused": Indicates the port is not open or Observo AI is not listening. Verify Socket Address, Port, and firewall settings.
"Authentication failed": Confirm the username and password match the Fluent output plugin configuration.
"Request too large": Check the receive buffer size setting; increase if incoming data exceeds the limit.
Monitor Logs and Data:
Verify data ingestion by monitoring the Analytics tab in the targeted Observo AI pipeline for data throughput.
Check Observo AI logs for errors or warnings related to data ingestion from the Fluent source.
Data not ingested
Incorrect address or port config
Verify Socket Address and Port settings
Authentication errors
Invalid or misconfigured credentials
Check username/password and Fluent config
Connectivity issues
Firewall or proxy blocking access
Test network connectivity and check firewall rules
"Connection refused"
Port not open or service not listening
Ensure Observo AI listens on correct address/port
"Authentication failed"
Mismatched credentials
Verify auth settings match Fluent output plugin
"Request too large"
Payload exceeds size limit
Increase receive buffer size in Advanced Settings
Resources
For additional guidance and detailed information, refer to the following resources:
Last updated
Was this helpful?