OpenTelemetry
The OpenTelemetry Source in Observo AI enables the ingestion of metrics, logs, and traces from OpenTelemetry clients or collectors via HTTP or gRPC, supporting real-time observability and analytics for distributed systems.
Purpose
The Observo AI OpenTelemetry source enables the ingestion of metrics, logs, and traces from OpenTelemetry (OTel) collectors or clients into the Observo AI platform for observability and analytics. It supports real-time telemetry data collection via HTTP or gRPC protocols, facilitating monitoring and insights across distributed systems. This integration allows organizations to centralize and analyze OTel data for enhanced visibility and troubleshooting.
Prerequisites
Before configuring the OpenTelemetry source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:
Observo AI Platform Setup:
The Observo AI Site must be installed and available.
Verify that the platform can process OTel data formats, including metrics such as Prometheus-style, logs, and traces, in Protocol Buffers (Protobuf) over HTTP or gRPC.
OpenTelemetry Client or Collector:
An OTel-compatible client or collector such as OpenTelemetry Collector, SDKs for supported languages like Java, Python, or Go configured to send telemetry data (OpenTelemetry Documentation).
The client must be instrumented to export metrics, logs, or traces to Observo AI’s ingestion endpoints using HTTP or gRPC protocols (OTel Collector Exporters).
Authentication:
Optional: If authentication is enabled, prepare an API key or token for the OTel client to include in HTTP/gRPC headers such as Authorization: Bearer <token>.
Ensure the token has permissions to send telemetry data to Observo AI.
TLS Certificates:
Optional: If TLS is enabled, prepare CA certificates, client certificates, and keys for secure communication.
Ensure the OTel client is configured to use TLS if required by Observo AI’s endpoints.
Network and Connectivity:
Ensure Observo AI’s ingestion endpoints (HTTP or gRPC) are accessible from the OTel client or collector.
Default ports:
HTTP: 4318 (OTLP/HTTP)
gRPC: 4317 (OTLP/gRPC)
Include authentication headers if enabled such as Authorization: Bearer <token>.
Enable TLS and provide certificates if required.
Configure firewalls or proxies to allow outbound traffic to Observo AI’s endpoints over these ports (OTel Protocol Specification).
Observo AI Platform
Must support OTel source
Verify data format compatibility
OTel Client/Collector
Sends metrics, logs, traces
Configure HTTP/gRPC exporters
Authentication
Optional API key or token
Include in headers if enabled
TLS Certificates
Optional for secure connections
Prepare CA/client certificates
Network
Connectivity to endpoints
Allow ports 4317 (gRPC), 4318 (HTTP)
Integration
The Integration section outlines the configurations for the OpenTelemetry source To configure OpenTelemetry as a source in Observo AI, follow these steps to set up and test the data flow:
Log in to Observo AI:
Navigate to Sources Tab
Click on “Add Sources” button and select “Create New”
Choose “OpenTelemetry” from the list of available destinations to begin configuration.
General Settings:
Name: A unique identifier for the source such as event-hubs-source-1.
**Description (**Optional): Description for the source
gRPC Address: Specify the gRPC server address in the format host:port. Please ensure this value is the same for gRPC and HTTP.
Example0.0.0.0:10000
HTTP Address: Specify the HTTP server address in the format host:port. Please ensure this value is the same for gRPC and HTTP.
Example0.0.0.0:10000
TLS Settings (Optional):
Enable TLS (False): Toggle to Enable for secure connections (recommended).
CA Certificate File: Path to the CA certificate file for validating the server's certificate. The certificate must be in the DER or PEM (X.509) format.
Example/path/to/ca.pem
Private Key File: Path to the client's certificate file. The certificate must be in DER, PEM (X.509), or PKCS#12 format.
Example/path/to/client.key
Private Key Password: Password for the private key file, if required. This has no effect unless key_file is set.
Example/path/to/client.key
Server Name for TLS Validation: Specify the server name for validating the server's certificate during TLS handshake. Only relevant for outgoing connections.
Verify Server Certificate (Enabled): Enable to verify the server's certificate is issued by a trusted CA. Certificates must not be expired and must be issued by a trusted issuer.
Verify Server Hostname (Enabled): Used to verify the server's hostname matches the certificate's hostname.
ALPN Protocols (Add): List of Application-Layer Protocol Negotiation (ALPN) protocols.
Example["h2"]
Advanced Settings:
Enable Acknowledgements (False): Enable or disable acknowledgements for telemetry data.
Parser Config:
Enable Source Log parser: (False)
Toggle Enable Source Log parser Switch to enable
Select appropriate Parser from the Source Log Parser dropdown
Add additional Parsers as needed
Pattern Extractor:
See Pattern Extractor for details.
Archival Destination:
Toggle Enable Archival on Source Switch to enable
Under Archival Destination, select from the list of Archival Destinations (Required)
Save and Test Configuration:
Save the configuration settings in Observo AI.
Verify that metrics, logs, or traces are being ingested from the OTel client.
Example Scenarios
MediCare Innovations, a fictitious healthcare technology company, integrates patient monitoring telemetry from OpenTelemetry collectors into Observo AI for real-time analytics and alerting. The configuration uses HTTP (telemetry.medicareinnovations.com:4318) and gRPC (telemetry.medicareinnovations.com:4317) endpoints with TLS enabled and acknowledgments for reliable ingestion of metrics, logs, and traces.
Standard OpenTelemetry Source Setup
Here is a standard OpenTelemetry Source configuration example. Only the required sections and their associated field updates are displayed in the table below:
General Settings
Name
otel-medicare-1
Unique identifier for the OpenTelemetry source, indicating MediCare Innovations’ telemetry ingestion.
Description
Ingest patient monitoring telemetry via OpenTelemetry for MediCare Innovations
Optional, provides context for the source’s purpose.
gRPC Address
telemetry.medicareinnovations.com:4317
gRPC server address, using the standard OTel port for OTLP/gRPC.
HTTP Address
telemetry.medicareinnovations.com:4318
HTTP server address, using the standard OTel port for OTLP/HTTP.
TLS Settings
Enable TLS
True
Enabled to require TLS for secure connections, critical for healthcare data.
CA Certificate File
-----BEGIN CERTIFICATE----- MIID... (PEM format)
Inline PEM string for the CA certificate, validating the server’s certificate.
Private Key File
-----BEGIN PRIVATE KEY----- MIIE... (PEM format)
Inline PEM string for the client’s private key, securely stored.
Private Key Password
MediCareKey2025!
Password to unlock the encrypted private key file, securely managed.
Server Name for TLS Validation
telemetry.medicareinnovations.com
Server name for validating the server’s certificate during TLS handshake.
Verify Server Certificate
True
Ensures the server’s certificate is valid and issued by a trusted CA.
Verify Server Hostname
True
Verifies the server’s hostname matches the certificate, enhancing security.
ALPN Protocols
["h2"]
Specifies HTTP/2 for gRPC, ensuring protocol compatibility.
Advanced Settings
Enable Acknowledgements
True
Enabled to ensure reliable telemetry data ingestion, critical for patient monitoring.
Parser Config
Enable Source Log Parser
True
Enabled to parse OTel data formats such as Protobuf for metrics, logs, traces.
Source Log Parser
OpenTelemetry Parser
Selected from the dropdown to match OTel data formats.
Additional Parsers
(None)
Not needed unless specific transformations are required.
Test Configuration:
Click “Save” to store the configuration settings in Observo AI.
Send sample telemetry data such as simulated patient vitals via an OTel client or collector to the configured endpoints. Verify ingestion by monitoring the Analytics tab in the Observo AI pipeline for metrics, logs, or traces.
Notes:
TLS Configuration: PEM certificate/key strings are placeholders; actual values must be provided by MediCare Innovations’ security team, securely stored in Observo AI. Enable TLS, Verify Server Certificate, and Verify Server Hostname are set to True for HIPAA-compliant security.
OTel Client: The OTel collector or client such as using Python/Java SDKs must be configured to export to http://telemetry.medicareinnovations.com:4318 (OTLP/HTTP) or telemetry.medicareinnovations.com:4317 (OTLP/gRPC), with matching TLS certificates.
Network: Ensure firewall rules allow traffic on ports 4317 (gRPC) and 4318 (HTTP) to telemetry.medicareinnovations.com. Proxies must be configured to allow these connections.
Troubleshooting: If issues occur such as “401 Unauthorized” or “Connection refused”, verify endpoint URLs, ports, and TLS certificates. Check OTel client logs and Observo AI’s Logs tab for errors, as per the Troubleshooting section.
Resources: Refer to OpenTelemetry Documentation, OpenTelemetry Collector Configuration, and OpenTelemetry Protocol Specification for guidance on configuring OTel exporters.
MediCare Innovations Context: This configuration supports real-time ingestion of patient monitoring telemetry, complementing other Observo AI sources for a unified healthcare observability pipeline, ensuring patient safety and system reliability.
Troubleshooting
If issues arise with the OpenTelemetry source in Observo AI, use the following steps to diagnose and resolve them:
Verify Configuration Settings:
Ensure Address, HTTP Port, and gRPC Port are correctly configured and match the OTel client’s exporter settings.
Confirm that the OTel client is sending data to the correct endpoint such as http://<observo-ai-host>:4318 for HTTP.
Check Authentication:
If authentication is enabled, verify that the API key or token is valid and included in the OTel client’s headers.
Ensure the token has permissions to send telemetry data.
Validate TLS Settings:
If TLS is enabled, confirm that CA, server, and client certificates are valid and correctly configured.
Check for certificate mismatches or expired certificates if “Validate Client Certs” is enabled.
Monitor Logs:
Check Observo AI’s Logs tab for errors or warnings related to data ingestion.
Use OTel Collector logs or client diagnostics to verify that data is being sent successfully (OTel Troubleshooting).
Validate Connectivity:
Ensure the OTel client can reach Observo AI’s endpoints on ports 4317 (gRPC) or 4318 (HTTP).
Check firewall rules or proxies to allow traffic to these ports.
Common Error Messages:
“401 Unauthorized”: Indicates an invalid or missing API key/token. Verify the token and headers in the OTel client.
“Connection refused”: Suggests incorrect port or firewall issues. Confirm ports 4317/4318 are open.
“No data ingested”: Check if the OTel client is sending data and that the endpoint URLs match Observo AI’s configuration.
Test Data Flow:
Capture real-time events and verify ingestion.
Use the Analytics tab in the targeted Observo AI pipeline to monitor data volume and ensure expected throughput
Data not ingested
Incorrect endpoint or port
Verify OTel client’s exporter URL
“401 Unauthorized”
Invalid/missing API key
Check token and headers
“Connection refused”
Firewall or wrong port
Allow ports 4317/4318
TLS errors
Certificate mismatch
Validate CA/server/client certificates
No data in source
OTel client not sending
Check OTel client logs and configuration
Resources
For additional guidance and detailed information, refer to the following resources:
Last updated
Was this helpful?