Prometheus Remote Write
The Prometheus Remote Write Source in Observo AI enables the ingestion of time-series metrics from a Prometheus instance or compatible client via the Remote Write API, supporting real-time monitoring, analysis, and alerting with secure HTTP/HTTPS communication.
Purpose
The purpose of the Observo AI Source Prometheus Remote Write is to enable users to ingest metrics data from a Prometheus instance or compatible client into the Observo AI platform for analysis and processing. It facilitates the collection of time-series metrics, typically in Protocol Buffers (protobuf) format, sent via the Prometheus Remote Write API, allowing organizations to streamline data pipelines, enhance observability, and support use cases such as monitoring, performance analysis, and alerting by processing Prometheus metrics in real time.
Prerequisites
Before configuring the Prometheus Remote Write source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:
Observo AI Platform Setup:
The Observo AI platform must be installed and operational, with support for the Prometheus Remote Write source.
Verify that the platform can process Prometheus metrics in the Protocol Buffers (protobuf) format, as delivered via the Remote Write API.
Prometheus Configuration:
A running Prometheus instance or compatible client must be configured to send metrics to Observo AI via Remote Write.
Obtain the endpoint URL for Observo AI (e.g., https://your-observo-instance:10090/prometheus) where metrics will be sent.
Authentication:
Prepare one of the following authentication methods:
Basic Authentication: Provide a username and password for HTTP Basic Auth, if required.
No Authentication: If no authentication is configured, ensure the Prometheus client can send data without credentials.
Network and Connectivity:
Ensure the Prometheus instance can communicate with the Observo AI endpoint over HTTP/HTTPS.
Check for firewall rules, proxy settings, or VPC configurations that may block traffic to the configured port (default: TCP 10090).
Observo AI Platform
Must be installed and support Prometheus Remote Write
Verify support for Prometheus metrics in protobuf format
Prometheus Config
Running Prometheus instance for metric submission
Configure Remote Write to send to Observo AI endpoint
Authentication
Basic Auth or no authentication
Prepare username/password if required; confirm client config
Network
Connectivity to Observo AI endpoint
Check firewalls, proxies, and VPC for port access (default: 10090)
Integration
The Integration section outlines the configurations for the Prometheus Remote Write source. To configure the Prometheus Remote Write source in Observo AI, follow these steps to set up and test the data flow:
Log in to Observo AI:
Navigate to the Sources tab.
Click the Add Source button and select Create New.
Choose Prometheus Remote Write from the list of available sources to begin configuration.
General Settings:
Name: A unique identifier for the source, such as prom-remote-write-source-1.
Description (Optional): Provide a description for the source.
Socket Address: Enter the hostname or IP address to listen for Prometheus Remote Write requests. Default: 0.0.0.0:10001
Example0.0.0.0:10000
Authentication Settings (Optional):
Username: Username if using basic authentication.
Password: Password if using basic authentication.
TLS Configuration (Optional):
TLS Enable (False): Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.
TLS CA File: Absolute path to an additional CA certificate file. The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format. Default: /etc/certs/ca.crt
Example/path/to/certificate_authority.crt
TLS Crt File: Absolute path to a certificate file used to identify this server. The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format. If this is set, and is not a PKCS#12 archive, key_file must also be set. Default: /etc/certs/tls.crt
Example/path/to/host_certificate.crt
TLS Key File: Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format. Default: /etc/certs/tls.key
TLS Key Pass: Passphrase used to unlock the encrypted key file. This has no effect unless key_file is set.
Examples${KEY_PASS_ENV_VAR}
PassWord1
TLS Verify Certificate (False): Enables certificate verification. If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate. Relevant for both incoming and outgoing connections. Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.
TLS Verify Hostname (True): Enables hostname verification. If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension. Only relevant for outgoing connections. Do NOT set this to false unless you understand the risks of not verifying the remote hostname.
Parser Config:
Enable Source Log Parser: (False)
Toggle Enable Source Log Parser Switch to enable.
Select appropriate Parser from the Source Log Parser dropdown.
Add additional Parsers as needed.
Pattern Extractor:
Refer to Observo AI's Pattern Extractor documentation for details on configuring pattern-based data extraction.
Archival Destination:
Toggle Enable Archival on Source Switch to enable.
Under Archival Destination, select from the list of Archival Destinations (Required).
Save and Test Configuration:
Save the configuration settings in Observo AI.
Configure your Prometheus instance to send metrics to the Observo AI endpoint (e.g., via remote_write in prometheus.yml).
Verify ingestion in the Analytics tab for data flow.
Example Scenarios
Nexus Service Solutions, a fictitious enterprise in the service provider sector, specializes in IT infrastructure management and customer support services. To enhance their monitoring capabilities, Nexus aims to ingest time-series metrics from their Prometheus instance into the Observo AI platform using the Prometheus Remote Write API. This integration will enable real-time performance analysis, alerting, and observability of their infrastructure. Below is the detailed configuration process for setting up the Prometheus Remote Write source in Observo AI, based on the provided documentation, with all required fields specified.
Standard Prometheus Remote Write Source Setup
Here is a standard Prometheus Remote Write Source configuration example. Only the required sections and their associated field updates are displayed in the table below:
General Settings
Name
prometheus-metrics
Unique identifier for the Prometheus Remote Write source.
Description
Source for ingesting Prometheus infrastructure metrics
Optional description for clarity.
Socket Address
0.0.0.0:10090
Listens for Prometheus Remote Write requests on port 10090 across all interfaces.
Authentication Settings
Username
nexus_admin
Username for HTTP Basic Authentication.
Password
${PROMETHEUS_PASSWORD}
Password stored securely in Observo AI’s secure storage.
TLS Configuration
TLS Enable
True
Enables TLS for secure communication.
TLS Ca File
-----BEGIN CERTIFICATE-----\nMIID...==\n-----END CERTIFICATE-----
Inline CA certificate in PEM format for Observo AI endpoint.
TLS Crt File
/certs/nexus_server_cert.pem
Path to the server certificate in PEM format.
TLS Key File
/certs/nexus_server_key.pem
Path to the server private key in PEM format.
TLS Key Pass
${TLS_KEY_PASSPHRASE}
Passphrase for the encrypted private key, stored securely.
TLS Verify Certificate
True
Enables certificate verification for secure connections.
TLS Verify Hostname
True
Verifies the hostname in the client’s TLS certificate.
Advanced Settings
Proxy URL
http://proxy.nexusservices.com:8080
Proxy server for outgoing connections, if applicable.
Read Buffer Size
32768
Read buffer size of 32 KB for HTTP responses.
Write Buffer Size
32768
Write buffer size of 32 KB for HTTP requests.
Timeout
15s
HTTP request timeout set to 15 seconds to handle metrics ingestion.
Compression
Snappy
Uses Snappy compression for fast processing of metrics data.
Test Configuration
Save the configuration in the Observo AI interface.
Configure the Prometheus instance to send metrics to https://observo.nexusservices.com:10090/prometheus using the remote_write section in prometheus.yml, with matching authentication credentials.
Verify metrics ingestion in the Observo AI Analytics tab, checking for expected metric throughput.
Monitor Observo AI logs for errors and confirm data flow from the Prometheus instance.
Scenario Troubleshooting
Authentication Errors: Ensure nexus_admin and ${PROMETHEUS_PASSWORD} match the credentials in the Prometheus remote_write configuration.
Connectivity Issues: Verify that port 10090 is open and the proxy at http://proxy.nexusservices.com:8080 allows traffic to observo.nexusservices.com. Test with curl or netcat.
Data Not Ingested: Confirm that Prometheus is correctly configured to send metrics in Protocol Buffers format and that Observo AI supports protobuf parsing.
Connection Refused: Check that Observo AI is listening on 0.0.0.0:10090 and that firewall rules allow incoming traffic.
Request Too Large: Increase buffer sizes in Advanced Settings if large metric payloads are causing issues.
This configuration enables Nexus Service Solutions to securely ingest Prometheus metrics into Observo AI, supporting real-time infrastructure monitoring and alerting.
Troubleshooting
If issues arise with the Prometheus Remote Write source in Observo AI, use the following steps to diagnose and resolve them:
Verify Configuration Settings:
Ensure fields like Address, Port, and Authentication settings match the Prometheus client’s remote_write configuration.
Confirm the port (default: 9090) is open and accessible.
Check Authentication:
For Basic Auth, verify the username and password are correct and match the credentials configured in the Prometheus remote_write settings.
Check Observo AI logs for authentication failure errors.
Validate Network Connectivity:
Ensure firewall rules, proxy settings, or VPC configurations allow traffic from the Prometheus instance to the Observo AI endpoint.
Test connectivity using tools like curl or netcat to the configured address and port.
Common Error Messages:
"Connection refused": Indicates the port is not open or Observo AI is not listening. Verify Address, Port, and firewall settings.
"Authentication failed": Confirm the username and password match the Prometheus client configuration.
"Request too large": Check the Max Body Size setting; increase if incoming metrics exceed the limit.
Monitor Logs and Data:
Verify data ingestion by monitoring the Analytics tab in the targeted Observo AI pipeline for metric throughput.
Check Observo AI logs for errors or warnings related to data ingestion from the Prometheus Remote Write source.
Data not ingested
Incorrect address or port config
Verify Address and Port settings
Authentication errors
Invalid or misconfigured credentials
Check username/password and Prometheus config
Connectivity issues
Firewall or proxy blocking access
Test network connectivity and check firewall rules
"Connection refused"
Port not open or service not listening
Ensure Observo AI listens on correct address/port
"Authentication failed"
Mismatched credentials
Verify auth settings match Prometheus remote_write
"Request too large"
Payload exceeds size limit
Increase Max Body Size in Advanced Settings
Resources
For additional guidance and detailed information, refer to the following resources:
External Documentation:
Last updated
Was this helpful?