Log Visualization and Testing Guide

Overview

The log visualization and testing system provides some useful functionalities to help you test your pipeline configuration before deployment and to visualize the transformation of your logs through different components in your pipeline. In this guide, we will cover the following functionalities:

  • Log Capture

  • Log Preview

  • Test Source/Transform

  • Log Replay

Log Capture

Capture real log samples from your running pipeline for testing and validation. You can capture logs from any component in your pipeline. This will help you to validate if your data is reaching the pipeline and if the pipeline is processing the data as expected. You can also use the captured logs to test your pipeline configuration before deployment.

Features

  • Capture live logs from any component

  • Specify number of lines to capture (default: 100 lines)

  • Set custom timeout duration (default: 5000ms)

  • Store samples for future use

  • Download captured log samples for offline analysis

  • Upload existing log samples for testing

Managing Log Samples

Log samples can be managed through the Settings > Files section of the Observo platform. This interface allows you to:

  • View all captured log samples

  • Download log samples for offline analysis

  • Upload custom log samples for testing

  • Preview log sample contents

  • Delete unnecessary log samples

Working with Log Samples

Downloading Log Samples

  1. Navigate to Settings > Files

  2. Locate the log sample you want to download

  3. Click on the file name and the file preview will be displayed

  4. Click on the download file icon displayed at the bottom right corner of the file preview

Download Log Sample

Uploading Log Samples

  1. In Settings > Files, click "Add File"

  2. Select "Log Sample" as the file type

  3. Choose your log file or copy and paste the log sample into the text area

  4. Add a description (optional)

  5. Click "Save" to save

Upload Log Sample

Using Samples in Log Preview

  1. In your pipeline view, select the component you want to test

  2. Click "Log Preview" to open the preview window

  3. Under "...", choose "Import Data"

  4. Choose the log sample you want to import

  5. Click on the "Set File" button

Log Preview Sample Selection
Log Preview Sample Set

Log Replay with Samples

  1. In the Log Preview window, click the "..." menu

  2. Choose your sample file or use the captured logs. The logs displayed at the out tab of the node will be replayed to the destination.

  3. Click on the "Play Logs" button to start the replay

Log Replay Configuration

Storage

  • Captured logs are stored with unique identifiers

  • Format: <source_name>_<source_id>_captured.log for source captures

  • Samples can be retrieved and managed through the API or UI

  • Access log samples through Settings > Files section

Log Preview

Log preview enables you to see how your logs are transformed as they pass through various components in your pipeline. This feature is particularly helpful for understanding how your logs and metrics are optimized by the pipeline components. You can preview the transformations using data captured from your live pipeline, previously saved data, or even data you have written yourself in the scratch pad of the Observo UI. The UI displays the incoming logs at each selected node, the transformations applied, and the outgoing logs after processing. Additionally, it provides a diff window to compare the incoming and outgoing logs. You can update any transform or source/sink configuration and can see the data preview before actually deploying the Pipeline.

How it works

  1. Select the nodes you want to preview

  2. Click on the Log preview toggle button

  3. Get immediate feedback showing:

    • Incoming logs at each selected node

    • How the logs are transformed

    • Outgoing logs after processing

    • A diff window to compare the incoming and outgoing logs

Notes

  • Preview is non-destructive and doesn't affect your production pipeline

Test Source/Transform

Test source/transform is almost the same as log preview. The only difference is that you can select the source/transform node to test. The main difference is that you can test the source/transform configuration on the fly and see the data preview before even placing the node in the pipeline's temporary cofiguration.

How it works

  1. Select the source/transform node you want to test

  2. Click on the Log preview toggle button. This will open the log preview window.

  3. Change the source/transform configuration as needed.

  4. Click on the Test button

  5. Get immediate feedback showing:

    • How the logs are transformed with the new source/transform configuration

    • A diff window to compare the impact of the new source/transform configuration

Log Replay

Log replay enables you to verify your pipeline configuration by replaying sample or captured logs to the actual destination using the current pipeline setup. This feature is especially useful for testing your pipeline, site, and destination configurations. It sends the data displayed in the preview window to the actual destination, allowing you to visualize and analyze the sample data at the real destination. Unlike log preview and other features, log replay sends data to the actual destination. This way, you can also test your network and destination configurations, ensuring that the pipeline can send data to the destination correctly and that the data format is accurate.

How it works

  1. Select The Node: Select the node you want to start replaying from

  2. Select Sample Logs

    • Choose the log samples you want to replay from the captured or saved logs

    • You can also upload your own log samples for replay

  3. Initiate Replay

    • Click on the Play Logs button to begin appeated after getting on the Log Preview window. The Play Logs button is located at the right top corner of the Log Preview window under ....

    • The system will send the selected logs to the destinations using the current pipeline configuration started from the selected node.

  4. Monitor Replay Progress

    • Track the progress of the replay start/success/failure through the user notification.

  5. Analyze Results

    • Once the replay is complete, review the results at the actual destinations

    • Compare the replayed logs with the expected output to verify the pipeline configuration

    • Use the results to make any necessary adjustments to your pipeline setup

Workflow

  1. Initiate Capture

    • Select components to capture from

    • Specify capture parameters (lines, timeout)

    • System sends capture event to control agent

  2. Monitor Progress

    • Receive notifications about capture status

    • View capture progress in real-time

  3. Access Results

    • Browse captured samples

    • Download for analysis

    • Use in pipeline testing

Best Practices

Log Preview

  • Test with representative sample data

  • Verify transformations before deployment

  • Check both success and error cases

  • Use preview for iterative development of transformations

Log Capture

  • Capture during peak activity periods

  • Store diverse sample sets

  • Label samples clearly for future reference

  • Regular capture of critical components

  • Error details when applicable

Notification Examples

  • "Sample log capture started for [100000000000012344]."

  • "Sample log capture succeeded for [100000000000012344]"

  • "Log replay started for Pipeline Id: 2832. Replay Id: replay-20250107163304-45364.

  • "Log replay succeeded for Pipeline ID: 2832, Replay ID: replay-20250107163304-45364"

  • "Log preview failed: Invalid transform configuration"

PreviousSentiment AnalyzerNextAccess Management

Last updated

Was this helpful?