Script

Script Source in Observo AI enables users to create custom data ingestion workflows using Python and bash scripts. This source allows for the execution of user-defined scripts to fetch, transform, and send data to Observo AI for analysis and processing. It provides flexibility in handling various data formats and sources, making it suitable for complex data ingestion scenarios.

Users can upload Python and bash scripts that execute periodically, enabling custom logic for data collection and processing workflows.

Purpose

The purpose of the Script Source in Observo AI is to enable users to execute custom Python and bash scripts for flexible data ingestion and collection workflows. This source allows organizations to implement tailored data retrieval logic by uploading and scheduling periodic execution of custom scripts. It facilitates the collection of data from diverse sources, complex data transformations, and custom processing pipelines that may not be covered by standard connectors. The Script Source empowers users to fetch data from proprietary systems, perform custom authentication flows, handle unique data formats, and implement specialized business logic for data collection, ultimately providing maximum flexibility for organizations to bring any type of data into Observo AI for comprehensive analysis, monitoring, and decision-making.

Prerequisites

Before configuring the Script source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:

  • Observo AI Platform Setup:

    • The Observo AI platform must be installed and operational, with support for the Script as a data source.

    • Verify that the platform supports common data formats such as JSON, CSV, or plain text. Additional formats may require specific parser configurations.

  • Network and Connectivity:

    • Ensure Observo AI can communicate with the HTTP/S endpoint being used in the script.

    • Check for proxy settings, firewall rules, or VPC endpoint configurations that may affect connectivity to the external endpoint.

Prerequisite
Description
Notes

Observo AI Platform

Must be installed and support HTTP Collector (Pull)

Verify support for JSON, CSV, etc.; additional parsers may be needed

HTTP/S Endpoint

Active endpoint for data submission

Obtain URL and credentials from the data provider

Network

Connectivity to the HTTP/S endpoint

Check VPC endpoints, proxies, and firewalls

Important: Secure and Use the Script Source Properly

Script execution in Observo AI runs within a secure jail environment to provide isolation and enhanced security. However, you should still exercise caution and follow security best practices when uploading and executing scripts.

  • Security Architecture:

    • Scripts execute within a sandboxed jail environment for isolation

    • The jail provides limited access to system resources and network connectivity

    • Script execution is contained to prevent unauthorized access to the host system

  • Security Considerations:

    • While jailed, scripts still require careful validation before execution

    • Malicious scripts could potentially consume system resources within the jail

    • Scripts may have limited but legitimate access to network resources for data collection

    • Both Python and bash scripts can perform operations within their sandboxed environment

  • Best Practices:

    • Always validate and thoroughly test your scripts before production use

    • Implement proper input validation and error handling in your scripts

    • Ensure scripts have defined execution timeouts and do not contain infinite loops

    • Review scripts for potential security vulnerabilities and resource consumption issues

    • Design scripts to be efficient and avoid excessive resource usage

  • Script Execution Guidelines:

    • Scripts must complete execution within reasonable time limits

    • Avoid infinite loops or blocking operations that could consume jail resources

    • Implement proper cleanup and resource management within scripts

    • Test script behavior under various failure scenarios

    • Ensure scripts gracefully handle network connectivity limitations within the jail

The jail environment provides an additional layer of security, but proper script development and testing practices remain essential for reliable and secure data collection workflows.

Integration

The Integration section outlines the configurations for the Script source. To configure the Script as a source in Observo AI, follow these steps to set up and test the data flow:

  1. Log in to Observo AI:

  • Navigate to the Sources tab.

  • Click the Add Source button and select Create New.

  • Choose Script from the list of available sources to begin configuration.

  1. General Settings:

  • Name: A unique identifier for the source, such as script-collector.

  • Description: (Optional): Provide a description for the source.

  • File: Upload a Python or bash script containing your custom data collection logic. The script should implement the necessary functionality to retrieve data from your target source and output the collected data to stdout for processing by Observo AI. Always validate and test your script thoroughly in a development environment before uploading to ensure proper functionality and security compliance.

    Examples

    python_script.py

    shell_script.sh

  • Environment Variables: (Optional): Environment variables to be used within the script. This will be available as system environment variables during script execution.

    Key (Default)
    Value (Default)

    API_TOKEN

    <API Token>

    BASE_URL

    https://api.example.com

  • Checkpoints: (Optional): Enable checkpoints to track the last successful data collection point. This is useful for incremental data collection. This will be the seed value for the first run.

    Key (Default)
    Value (Default)

    PAGE

    1

    StartTime

    2025-08-01T00:00:00Z

  • Command: Command to be used to execute the script. Choose between python3 for Python scripts or bash for shell scripts.

    Examples

    Python3

    bash

    • Time in seconds to pause between script executions. (Optional): Duration between consecutive data collection requests. Default: 1m

      Examples

      60s

      100s

    • Lua Script (Modify as needed): Lua script to transform the data collected by the main script. This is the place to modify the checkpoint value if needed and to transform the data into the expected format.

      function start(config)
  exec{
    params = {
        cmd = config.cmdline,
        env = config.env,
        work_dir = "/tmp",
        stdout = true,
        stderr = true
    },
    fn = "data",
    retry = false
  }
end

function data(config, response)
 emit{log = { ret = response.ret, message = response.out}}
end
PreviousSalesforce Event MonitoringNextServiceNow

Last updated

Was this helpful?