Edge Collector

The Observo AI Edge Collector, as detailed in the blog post - Feature Deep Dive: Observo AI Edge Collector and Fleet Management at Scale provides a robust solution for large-scale telemetry data management, with a strong emphasis on our architecture and agent installation capabilities. Our fleet management is a first-class capability, enabling centralized control and seamless scalability across thousands of edge nodes. This architecture simplifies deployment, configuration, and maintenance, ensuring consistent performance and rapid updates across distributed environments. By treating fleet management as a core component, Observo AI empowers security and DevOps teams to efficiently handle growing data volumes while maintaining operational agility.

Our agent installation process is designed to be declarative, repeatable, and cross-platform, streamlining onboarding and ensuring compatibility across diverse systems. This approach allows for consistent and automated deployments, reducing setup complexity and enabling teams to focus on data optimization. The Edge Collector supports parsing, transforming, reducing, and enriching telemetry data at the edge, cutting costs by up to 80% and improving MTTR by over 40%. With integrations for 50+ sources and destinations, it ensures flexibility and compliance, making it a critical tool for enterprises like Bill and Informatica processing petabytes daily.

Key Benefits

Centralized Management: Configuration updates are pushed from a central UI, allowing you to:

• Roll out changes instantly across thousands of agents

• Version control and audit config changes

• Deploy by fleet, platform, or site

Edge Filtering: One of the biggest cost drivers in cloud-native telemetry is the ingestion of unnecessary data. The Edge Collector allows teams to apply filters and exclusion rules directly at the edge, ensuring that only high-value data enters the pipeline. Examples include:

• Drop 90% of health-check logs from Kubernetes clusters

• Forward only audit events, not debug logs, from Windows hosts

• Collect metrics only during business hours to reduce noise

Cross-Platform Support: Installing Observo AI's Edge Collector is designed to be frictionless. Once a configuration is created in the UI, the platform auto-generates an installation script tailored to the platform (Linux, Windows, macOS) and associated site.

Standards-Based: Edge Collector supports the use of OTel collector schemas (including the OTLP protocol), allowing for alignment with broader observability strategies, which enables:

• Schema normalization before data enters the pipeline

• Easy integration with standards-based platforms

• Reduction of vendor lock-in due to shared field formats

This diagram (Figure 2) illustrates the Observo AI Edge Collector (Agent) architecture for scalable telemetry data management, powered by OpenTelemetry (OTel) standards. A centralized Fleet Manager (Console) orchestrates three fleets – Prod East, Prod Auth (for user authentication), and Prod West – each collecting logs, traces, and metrics from cross-platform edge agents (Windows, Linux, macOS). Edge agents optimize, normalize, and enrich telemetry data using OTel schemas and route telemetry data to two Observo sites: Prod East (serving Prod East and Prod Auth fleets) and Prod West (serving Prod West and Prod Auth fleets). At the Observo Sites, Sources collect telemetry data that feed multiple pipelines—each capable of supporting numerous Destinations. These pipelines optimize, normalize, and enrich ingested data through transforms such as functions, parsers, serializers, and optimizers, while also applying data insights, pattern matching and sentiment analysis. Together, these capabilities enable seamless routing to more than 50 destination types. This visual highlights centralized configuration management, seamless cross-platform deployment, flexible multi-destination routing, and scalable operations, achieving up to 80% data reduction and over 40% improvement in MTTR.

Figure: Edge Collector (Agent) Architecture

Replacement for Legacy Solutions

Edge Collector is a direct replacement for:

• rsyslog, syslog-ng: with flexible parsing and schema alignment

• Fluent Bit / Logstash: with simpler configuration and better scale

• Proprietary agents, heavy forwarders or installed collectors from legacy SIEM vendors that leads to significantly lower CPU footprint and no proprietary lock-in

The Observo AI Edge Collector capability offers advantages over legacy SIEM agents like Splunk Heavy Forwarders and Sumo Logic Installed Collectors such as:

1. Lower CPU Footprint: Uses AI-driven data optimization to reduce CPU usage to <1% by filtering and compressing data at the edge, unlike resource-heavy Splunk Heavy Forwarders.

2. No Proprietary Lock-In: Supports open formats such as Parquet, Syslog and flexible routing to any destination, avoiding Splunk and Sumo Logic’s vendor-specific ecosystems.

3. Reduced Data Volume: Cuts data by up to 80% through intelligent filtering, lowering processing demands compared to unfiltered data sent by legacy agents.

4. Cost Efficiency: Stores data in low-cost formats like Parquet, reducing reliance on expensive proprietary storage and cutting costs by up to 50%.

5. Scalable Fleet Management: Centralizes configuration and updates for thousands of nodes, simplifying operations compared to manual management of Splunk or Sumo Logic agents.

6. Enhanced Data Quality: Enriches data with context such as Geo-IP and removes noise at the edge, improving analytics efficiency over legacy agents’ raw data forwarding.

Fleet Visibility and Control

All agents are visible in the Fleet dashboard with metrics such as:

• Status (active, inactive)

• Host OS, version, IP, MAC address

• CPU/memory utilization

• Volume of telemetry sent

• Config version

This structure supports rapid onboarding at scale—one script, many nodes. Observo AI's Edge Collector installation script is built for fleet-wide deployment with minimal manual intervention. Whether you're managing ten nodes or ten thousand, the same lightweight script can be executed across all of them—automatically configuring each instance to collect and route data based on predefined settings.