Dynatrace

This document provides step-by-step instructions for configuring Observo AI's HTTP destination to ingest logs into Dynatrace using the Dynatrace Log Monitoring API v2. The integration enables secure, scalable delivery of observability data from Observo AI pipelines to Dynatrace for centralized log management and analytics.

Purpose

The Observo AI Dynatrace HTTP Destination enables organizations to route telemetry data (logs, metrics, traces) from Observo AI Sites directly to Dynatrace environments for comprehensive monitoring and analysis. Key benefits include:

  • Unified Log Management: Centralize logs from multiple sources into Dynatrace for correlation with application performance metrics.

  • Real-time Analytics: : Leverage Dynatrace's AI-powered analytics and alerting capabilities on your observability data.

  • Data Enrichment: Utilize Observo AI's pipeline capabilities to filter, mask, and enrich data before forwarding to Dynatrace.

  • Scalable Delivery:: Support for high-volume log ingestion with batching and retry mechanisms.

Prerequisites

Before configuring an HTTP destination in Observo AI, ensure the following requirements are met:

  • Observo AI Account: You must have an active Observo AI account with administrative access to the Observo console.

  • Active Dynatrace Environment: SaaS environment or on-premises Managed deployment.

  • Access Token for Authentication: Dynatrace access token with logs.ingest scope.

  • Environment ID: Your Dynatrace environment identifier.(e.g. abc12345 from https://abc12345.live.dynatrace.com)

  • ActiveGate(Optional): For on-premises deployments or when using Environment ActiveGate endpoint.

  • Network Access: Ensure your Observo Site (data plane) has network connectivity to Dynatrace endpoint, with no firewall rules blocking outbound traffic.

  • Data Schema: Understand the expected JSON schema or data format required by Dynatrace to avoid integration issues.

  • Observo Site Deployment: A functional Observo Site must be deployed in your environment (on-premises or cloud) to handle data routing. Refer to the Observo AI documentation for deployment instructions.

  • Log attributes Read the official Dynatrace documentation thoroughly to understand the defined log attributes.

Prerequisite
Description
Notes

Observo AI Platform

The Observo AI Site must be installed and available.

Verify support for JSON/text payloads

Dynatrace Environment

Active SaaS or Managed environment

Obtain environment ID from URL

Network

Connectivity to HTTP/HTTPS ports

Default ports: TCP 80 (HTTP), TCP 443 (HTTPS); check firewall/proxy

Access Token

Access token with logs.ingest scope

Store securely and rotate regularly

Dynatrace API Token Configuration

Creating an API Token:

  • Access Token Management:

    • Log into your Dynatrace environment.

    • Navigate to Access tokens via search bar

    • Click Generate new token.

  • Token Configuration:

    • Name: Provide a descriptive name (e.g. observo-log-ingest-token)

    • Scopes: Select log.ingest scope.

    • Description: Add context about the token usage

  • Token Format: Dynatrace uses a three-component token format separated by dots: dt0c01.ABC123DEF456.GHIJKLMNOPQRSTUVWXYZ123456789

    • Prefix: dt0c01 identifies the token type

    • Public ID: ABC123DEF456 (24-character identifier)

    • Secret: GHIJKLMNOPQRSTUVWXYZ123456789 (secret portion)

Integration

Follow these steps to configure the HTTP destination in Observo AI:

  • Log in to Observo AI:

    • Navigate to the Destinations tab.

    • Click the Add Destinations button and select Create New.

    • Choose "HTTP" from the list of available destinations.

  • General Settings:

    • Name: Unique identifier for the destination. Example: dynatrace-logs-destination

    • Description (Optional): Provide a description for the destination.

    • URL / URI: Dynatrace log ingest endpoint.

    • Choose the appropriate endpoint based on your Dynatrace deployment:

      • For SaaS Environment: https://{your-environment-id}.live.dynatrace.com/api/v2/logs/ingest

      • For ActiveGate Environment : https://{your-activegate-domain}:9999/e/{your-environment-id}/api/v2/logs/ingest

    • HTTP Method: Required method for Dynatrace API. Default: POST

  • Encoding:

    • Encoding Codec: The codec to use for encoding events. Dynatrace accepts JSON format.

    • Pretty JSON: Compact JSON for efficiency. Example: False

    • Timestamp Format: ISO 8601 format preferred by Dynatrace. Example: RFC3339 format

  • Request Configuration:

    • Request Headers: Custom HTTP request headers to include with each request.Dynatrace expect the following headers in the request:

      Key
      Value

      Authorization

      Api-Token

      Content-Type

      application/json; charset=utf-8

    • Request Concurrency: Configuration for outbound request concurrency. Default: Adaptive concurrency

    • Time window for rate limiting in seconds: Default: 1

    • Max requests allowed in time window: (Empty)

    • Max retries for failed requests: (Empty)

    • Min wait time before first retry in seconds: (Empty)

    • Max wait time between retries in seconds: Default: 3600

    • Max time in seconds after which a request is considered as failed: Default: 60

  • Batching Configuration:

    • Batch Max Bytes: Set the maximum batch size in bytes. Example: 1048576.

    • Batch Max Events: Set the maximum number of events per batch. Optimal batch size for Dynatrace is 100.

    • Batch Timeout Seconds: Set the maximum time to wait before sending a batch. Example: 5

  • Acknowledgement:

    • Enable Acknowledgements (False)

  • Framing:

    • Framing Method: Standard JSON streaming format. Example: Newline Delimited

  • TLS Configuration (Optional):

    • TLS Verify Certificate (True): Enables certificate verification. Certificates must be valid in terms of not being expired, and being issued by a trusted issuer. Verify Dynatrace SSL certificate.

    • TLS Verify Hostname (True): Enables hostname verification. Verify hostname matches certificate.

  • Advanced Settings:

    • Compression: Enable compression to reduce data transfer size.

      Example

      Gzip

  • Save and Test Configuration:

    • Save the configuration settings in Observo AI.

    • Send sample trace data and verify that it appears in the HTTP-based service.

Example Configuration

Category
Field
Value

General Settings

Name

dynatrace-logs

Description

Log ingestion to Dynatrace

URL/URI

https://abc12345.live.dynatrace.com/api/v2/logs/ingest

HTTP Method

POST

Encoding

Encoding Codec

JSON Encoding

Pretty JSON

Disabled

Format of event timestamps when encoding

RFC3339 format

Request Configuration

Request Headers

Authorization : Api-Token dt0c01.UXLPKLMK2232FBZL4ZB22SCX.OOS2VLJWJNMMBFGNELBP6EBQIDFB5Q253GKQURO7T44ZA3ZPSU96VWF3FFIJKOIB Content-Type : application/json; charset=utf-8

Request Concurrency

Adaptive concurrency

Time window for rate limiting in seconds

1

Max wait time between retries in seconds

3600

Max time in seconds after which a request is considered as failed

60

Batching Configuration

Max bytes in batch

1048576

Max events in batch

100

Batch timeout in seconds

5

Framing

Framing Method

Newline Delimited

Framing Character Delimited Delimiter

,

TLS Configuration

TLS Verify Certificate

Enabled

TLS Verify Hostname

Enabled

Data Format Requirements

  • Supported Log Formats: Dynatrace accepts logs in the following JSON structure

[
  {
    "content": "Application error occurred in user authentication module",
    "timestamp": "2024-08-08T15:30:45.123Z",
    "severity": "error",
    "log.source": "/var/log/application/auth.log",
    "service.name": "authentication-service",
    "host.name": "web-server-01",
    "custom.attribute": "custom value"
  }
]

  • Required Fields::

    • content: The log message content

  • Optional but Recommended Fields:

    • timestamp: Event timestamp (RFC3339, UTC milliseconds, or RFC3164)

    • severity: Log level (debug, info, warn, error, fatal)

    • log.source: Source file or system

    • host.name: Originating host

    • service.name: Service identifier

Troubleshooting

Common issues and solutions when configuring an HTTP destination:

  • Connection Timeout Errors:

    • Issue: The Observo Site cannot connect to the HTTP endpoint.

    • Solution: Verify the URL is correct and accessible. Check network connectivity and firewall rules to ensure outbound traffic to the endpoint is allowed. Test the endpoint using tools like curl or Postman. Adjust request timeout settings.

  • Authentication Errors:

    • Issue: The endpoint rejects requests due to invalid credentials.

    • Solution: Verify API token has logs.ingest scope. Ensure the token is valid and not expired. Contact the token provider for updated credentials if needed.

  • Data Format Issues:

    • Issue: The endpoint returns errors due to incorrect data format or schema.

    • Solution: Verify timestamp format (use RFC3339). Verify timestamp format (use RFC3339). Use Observo’s transform capabilities to structure or enrich data to match the required format.Review Dynatrace attribute flattening rules

  • Rate Limiting:

    • Issue: The endpoint returns HTTP 429 (Too Many Requests) errors.

    • Solution: Implement exponential backoff retry strategy. Reduce request concurrency. Increase batch timeout to send larger, less frequent batches.

  • Pipeline Failures:

    • Issue: The pipeline fails to deliver data to the HTTP destination.

    • Solution: Check the pipeline configuration for errors in source or transform settings. Review Observo logs for detailed error messages and ensure the Observo Site is operational.

Issue
Possible Cause
Resolution

No data forwarded

Incorrect IP/port or network block

Verify client configuration and network connectivity

Invalid payload format

Unsupported data format

Check JSON structure validity

Payload too large

HTTP 413 errors or payload size warnings

Reduce Batch Max Events (recommended: 50-100).Enable Gzip compression.

TLS errors

Certificate or version mismatch

Check TLS settings and certificate files

Authentication errors

Incorrect credentials or token

Verify token validity and confirm API token has logs.ingest scope

Resources

For additional guidance and detailed information, refer to the following resources:

PreviousDatadog TracesNextElasticsearch

Last updated

Was this helpful?