Datadog

This source enables you to ingest data from a Datadog Agent into Observo. Datadog agents are popular log aggregators and forwarding agents, and this integration enables you to ingest logs into the Observo platform for centralized observability and monitoring.

Purpose

The Observo AI Datadog source enables the ingestion of metrics, logs, and traces from Datadog agents or APIs into the Observo AI platform for centralized observability and analytics. It allows organizations to process and analyze Datadog data for monitoring, troubleshooting, and deriving insights. This integration facilitates seamless data collection from Datadog’s monitoring services to enhance visibility across systems.

Prerequisites

Before configuring the Datadog source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:

• Observo AI Platform Setup:

  • The Observo AI Site must be installed and operational.

  • Verify Datadog data formats, such as JSON for metrics, logs, and traces.

• Datadog Account:

  • An active Datadog account with access to the Datadog API (Datadog API Overview).

  • Obtain an API key and an application key from the Datadog platform:

    • API Key: Navigate to Organization Settings > API Keys in Datadog to create or retrieve an API key (Create API Key).

    • Application Key: Navigate to Organization Settings > Application Keys to create or retrieve an application key (Create Application Key).

  • Identify the appropriate Datadog site for your region such as datadoghq.com for US1, us3.datadoghq.com for US3, datadoghq.eu for EU, etc. (Datadog Sites).

• Permissions:

  • Ensure the API and application keys have sufficient permissions to access the desired data such as metrics, logs, traces.

  • For metrics, logs, or traces, the keys must have read access to the respective endpoints (Datadog API Permissions).

• Network and Connectivity:

  • Ensure Observo AI can communicate with Datadog API endpoints over HTTPS (port 443).

  • If using firewalls or proxies, configure them to allow outbound traffic to the appropriate Datadog site such as api.datadoghq.com, api.us3.datadoghq.com, etc..

  • No HTTP proxies are required, as the integration uses direct HTTPS connections.

Prerequisite	Description	Notes
Observo AI Platform	Must support Datadog source	Verify data format compatibility
Datadog Account	Active account with API access	Create API and application keys
Permissions	Read access to metrics/logs/traces	Ensure keys have correct scopes
Network	HTTPS connectivity to Datadog	Allow port 443, no HTTP proxies

Configure Datadog Agent to Forward Logs (Kubernetes)

To configure the Datadog Agent on your Kubernetes cluster to forward logs to Observo, follow these steps:

1. Create a values.yaml file and enter the required information for the Datadog Agent:

   configuration:
   datadog:
     site: "datadoghq.com"
     apiKey: "<Datadog API KEY>"
     logs:
       enabled: true
       containerCollectAll: true
   
   agents:
     # Use this for EKS
     podSecurity:
       apparmor:
         enabled: false
   
     customAgentConfig:
       logs_enabled: true
       skip_ssl_validation: true
       observability_pipelines_worker:
         logs:
           enabled: true
           url: "http://<OBSERVO SOURCE URL>"
   
     useConfigMap: true

   Replace <Datadog API KEY> with your Datadog API key and <OBSERVO SOURCE URL> with the appropriate URL for your Observo endpoint.

2. If you want TLS (Transport Layer Security) enabled, there are a few extra steps and configurations:

   • Download the Observo CA file from the Settings->Site page and save it in a local file, for example, ob_ca.cert.

   • Run the following command to create a Kubernetes secret to store the CA certificate: kubectl create secret -n datadog generic ca-certificate --from-file=ob_ca.crt

   • Update the values.yaml file with the following additional configurations:

     datadog:
       site: "datadoghq.com"
       apiKey: "<Datadog API KEY>"
       logs:
         enabled: true
         containerCollectAll: true
     
     agents:
       # Use this for EKS
       podSecurity:
         apparmor:
           enabled: false
     
       customAgentConfig:
         logs_enabled: true
         observability_pipelines_worker:
           logs:
             enabled: true
             url: "https://<OBSERVO SOURCE URL>"
     
       useConfigMap: true
       volumes:
         - name: ca-certificate
           secret:
             secretName: ca-certificate
       volumeMounts:
         - name: ca-certificate
           mountPath: /etc/ssl/certs/ob_ca.crt
           subPath: ca.crt
           readOnly: true

3. Run the following command to update the Datadog Agent daemon on your Kubernetes cluster:

   helm upgrade datadog datadog/datadog --namespace datadog --values ./values.yaml

These steps will modify and configure the Datadog Agent to send logs to your Observo endpoint. Make sure to replace the placeholder values (<Datadog API KEY>, <OBSERVO SOURCE URL>) with the actual values relevant to your setup.

Integration

The Integration section outlines default configurations. To configure Datadog as a source in Observo AI, follow these steps to set up and test the data flow:

1. Log in to Observo AI:

   • Navigate to Sources Tab

   • Click on “Add Source” button and select “Create New”

   • Choose “Datadog” from the list of available destinations to begin configuration.

2. General Settings:

   • Name: A unique identifier for the source such as datadog-source-1.

   • Description (Optional): Description for the source

   • Address: The socket address to accept connections on. It must include a port.

     Example
     0.0.0.0:80
     localhost:80

3. Framing (Optional):

   • Framing Delimiter (Empty): The character that delimits byte sequences.

   • Framing Max Length (None): The maximum length of the byte buffer. This length does not include the trailing delimiter. By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases. If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This will ensure that processing is not truly unbounded.

   • Framing Method (Empty): The framing method.

     Options
     Byte Frames
     Character Delimited
     Length Delimited
     Newline Delimited
     Octet Counting

   • Framing Newline Delimited Max Length (None): The maximum length of the byte buffer. This length does not include the trailing delimiter. By default, there is no maximum length enforced. If events are malformed, this can lead to additional resource usage as events continue to be buffered in memory, and can potentially lead to memory exhaustion in extreme cases. If there is a risk of processing malformed data, such as logs with user-controlled input, consider setting the maximum length to a reasonably large value as a safety net. This will ensure that processing is not truly unbounded.

   • Framing Octet Counting Max Length (None): The maximum length of the byte buffer.

4. TLS Configurations (Optional):

   • TLS Enabled (False): Whether or not to require TLS for incoming or outgoing connections. When enabled and used for incoming connections, an identity certificate is also required. See tls.crt_file for more information.

   • TLS CA File: Absolute path to an additional CA certificate file. The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.Default: /etc/certs/ca.crt

     Example
     /path/to/certificate_authority.crt

   • TLS Crt File (Empty): Absolute path to a certificate file used to identify this server. The certificate must be in DER, PEM (X.509), or PKCS#12 format. If this is set, and is not a PKCS#12 archive, key_file must also be set. Default: /etc/certs/tls.crt

     Example
     /path/to/host_certificate.crt

   • TLS Key File: Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Default: /etc/certs/tls.key

     Example
     /path/to/host_certificate.key

   • TLS Key Pass (Empty): Passphrase used to unlock the encrypted key file. This has no effect unless key_file is set.

     Examples
     ${KEY_PASS_ENV_VAR}
     PassWord1

   • TLS Verify Certificate (False): Enables certificate verification. If enabled, the hostname used to connect to the remote host must be present in the TLS certificate presented by the remote host, either as the Common Name or as an entry in the Subject Alternative Name extension. Only relevant for outgoing connections. Do NOT set this to false unless you understand the risks of not verifying the remote hostname.

   • TLS Verify Hostname (False): Enables hostname verification. If enabled, certificates must not be expired and must be issued by a trusted issuer. This verification operates in a hierarchical manner, checking that the leaf certificate (the certificate presented by the client/server) is not only valid, but that the issuer of that certificate is also valid, and so on until the verification process reaches a root certificate. Relevant for both incoming and outgoing connections. Do NOT set this to false unless you understand the risks of not verifying the validity of certificates.

5. Advanced Settings:

   • Disable Logs (False): If this is set to true, logs won't be accepted by the component.

   • Disable Metrics (False): If this is set to true, metrics won't be accepted by the component.

   • Disable Traces (False): If this is set to true, traces won't be accepted by the component.

   • Store Api Key (True): If this is set to true, when incoming events contain a Datadog API key, it will be stored in the event metadata and will be used if the event is sent to a Datadog sink.

6. Parser Config:

   • Enable Source Log parser: (False)

   • Toggle Enable Source Log parser Switch to enable

     • Select appropriate Parser from the Source Log Parser dropdown

     • Add additional Parsers as needed

7. Pattern Extractor:

   • See Pattern Extractor for details.

8. Archival Destination:

   • Toggle Enable Archival on Source Switch to enable

   • Under Archival Destination, select from the list of Archival Destinations (Required)

9. Save and Test Configuration:

   • Save the configuration settings.

   • Verify that data (metrics, logs, or traces) is being ingested from Datadog.

Note: For optimal performance, avoid enabling all data types (Metrics, Logs, Traces) in a single source to prevent excessive API load. Consider creating separate sources for each data type if needed.

Example Scenarios

MediCareTech Solutions, a fictitious healthcare technology enterprise, provides telemedicine platforms, patient management systems, and health analytics services. To enhance system reliability and monitor application performance, MediCareTech integrates the Observo AI platform to ingest logs and metrics from Datadog, focusing on application logs to track system health and user interactions. The IT team configures the Datadog source to collect JSON-formatted logs from their telemedicine platform, enabling centralized observability, troubleshooting, and compliance with healthcare data regulations. This setup allows them to validate data pipelines and derive insights in a secure environment.

Standard Datadog Source Setup

Here is a standard Datadog Source configuration example. Only the required sections and their associated field updates are displayed in the table below:

General Settings

Field	Value	Notes
Name	datadog-medicaretech-logs-1	Unique identifier for the Datadog source, specific to MediCareTech’s log collection.
Description	Application logs from Datadog for telemedicine platform monitoring	Optional description to clarify the purpose of the source.
Address	0.0.0.0:443	Socket address to accept HTTPS connections, using port 443 for secure communication.

Framing

Field	Value	Notes
Framing Delimiter	\n	Newline character as the delimiter for byte sequences, aligning with the Newline Delimited method.
Framing Max Length	8192	Set to 8192 bytes to limit buffer size, preventing memory issues with malformed logs.
Framing Method	Newline Delimited	Selected as specified to frame logs using newline delimiters.
Framing Newline Delimited Max Length	8192	Set to 8192 bytes to ensure consistent buffer limits for newline-delimited logs.
Framing Octet Counting Max Length	8192	Set to 8192 bytes for consistency across framing configurations, though not used in this method.

TLS Configuration

Field	Value	Notes
TLS Enabled	True	Enables TLS for secure incoming connections, required for healthcare data compliance.
TLS CA File	/etc/certs/ob_ca.crt	Path to the CA certificate in PEM format for verifying Datadog server certificates.
TLS Crt File	/etc/certs/tls_medicaretech.crt	Path to the server certificate in PEM format to identify the Observo AI server.
TLS Key File	/etc/certs/tls_medicaretech.key	Path to the private key in PEM format for secure connections.
TLS Key Pass	MediCare2025	Passphrase to unlock the encrypted key file for secure TLS setup.
TLS Verify Certificate	True	Enables certificate verification to ensure the Datadog server’s certificate is valid.
TLS Verify Hostname	True	Enables hostname verification to ensure the certificate matches api.datadoghq.com.

Advanced Settings

Field	Value	Notes
Disable Logs	False	Ensures logs are accepted, as this source focuses on application logs.
Disable Metrics	True	Disables metrics to focus solely on logs, reducing API load.
Disable Traces	True	Disables traces to optimize performance for log ingestion.
Store Api Key	True	Stores the Datadog API key in event metadata for use with a Datadog sink if needed.

Troubleshooting

If issues arise with the Datadog source in Observo AI, use the following steps to diagnose and resolve them:

• Verify Configuration Settings:

  • Ensure the Datadog Site, API Key, Application Key, and Data Types are correctly configured.

  • Confirm that the selected Datadog site matches your account’s region (such as datadoghq.com for US1, datadoghq.eu for EU.

• Check Authentication:

  • Verify that the API and application keys are valid and not revoked (Manage API Keys).

  • Ensure the keys have permissions to access the selected data types (metrics, logs, traces) (Datadog RBAC).

• Monitor Logs:

  • Check Observo AI’s Logs tab for errors or warnings related to data ingestion.

  • Use Datadog’s API logs or dashboard to verify API requests and responses (Datadog API Monitoring).

• Validate Connectivity:

  • Ensure Observo AI can reach Datadog API endpoints over HTTPS (port 443).

  • Check for firewall rules or proxies that may block access to the Datadog site such as api.datadoghq.com.

• Common Error Messages:

  • “Invalid API key” or “Invalid application key”: Indicates incorrect or revoked keys. Regenerate keys in Datadog and update the configuration.

  • “403 Forbidden”: Suggests insufficient permissions. Verify the keys’ scopes for metrics, logs, or traces.

  • “No data ingested”: Confirm that data is available in Datadog for the selected data types and that the Filename Filter is not overly restrictive.

• Test Data Flow:

  • Capture real-time events and verify ingestion.

  • Use the Analytics tab in the targeted Observo AI pipeline to monitor data volume and ensure expected throughput

Issue	Possible Cause	Resolution
Data not ingested	Incorrect Datadog Site or filter	Verify site and Filename Filter
“Invalid API key”	Revoked or incorrect key	Regenerate and update API key
“403 Forbidden”	Insufficient permissions	Check key scopes for data types
Connectivity issues	Firewall or proxy blocking	Allow HTTPS on port 443
No data in source	No data in Datadog	Confirm data availability in Datadog

Resources

For additional guidance and detailed information, refer to the following resources:

• Datadog API Overview: Comprehensive guide to Datadog’s API capabilities

• Datadog Sites: List of Datadog regional endpoints

• Create API and Application Keys: Instructions for generating API and application keys

• Datadog RBAC Permissions: Guide to managing permissions for API access

• Datadog API Monitoring: Instructions for monitoring API activity