Multi-Transform Optimizer
Observo AI leverages MultiTransform Optimizer to streamline data integration by intelligently combining multiple transforms, applying best practices tailored to specific source types. This approach ensures seamless standardization and processing of diverse datasets, enabling efficient handling of logs, metrics, and traces. By stitching together Observo AI transforms, optimize data pipelines, enhancing compatibility with external systems while maintaining consistency, scalability, and performance across security and observability workflows.
Purpose
Data comes in all sizes and shapes, and handling it effectively requires a robust framework for normalization and processing. The Observo AI platform offers the industry's most comprehensive set of data transforms, enabling seamless integration and standardization of diverse datasets. With advanced techniques, it adapts to varying data structures, from unstructured text to complex numerical inputs. Whether you’re working with security or observability data such as logs, metrics, or traces, Observo AI ensures that the data is processed consistently and efficiently.
Observo AI brings these elements together with a powerful platform that emphasizes both data security and observability. It enables seamless monitoring of data flows, ensuring that any security anomalies or potential risks are flagged immediately. At the same time, its comprehensive analytics tools allow businesses to gain deeper visibility into their data patterns, behaviors, and trends.
An Observo AI MultiTransform Optimizer utilizes the complete plethora of its Transforms to accomplish this goal:
Functions are in-line transforms that modify event data by adding, removing, or enriching fields to tailor logs to your specific needs.
Parsers convert raw or semi-structured log data into a structured format by extracting key information based on known patterns.
Serializers then format this structured data into standardized outputs that are compatible with external systems, such as SIEMs or other analytics tools.
Optimizers work to reduce data volume by aggregating, summarizing, or filtering events to improve processing efficiency and lower storage costs.
Sentiment Analyzer allows Security and DevOps teams to reduce alert fatigue and prioritize incidents by using AI to detect anomalies and assign sentiment scores to telemetry data.
MultiTransform Optimizers define a set of best practices that transform data, parsers structure it, serializers prepare it for downstream use, and optimizers ensure performance and scalability.
Current MultiTransform Optimizer Suite
Cisco ASA Optimizer: A curated set of AI-driven transforms that streamlines Cisco ASA firewall log data by filtering non-critical events such as event IDs 106011, 106013, 106015, sampling high-volume events such as 106100, 106023, and enriching data for enhanced security, seamless SIEM integration, and reduced costs.
Palo Alto Optimizer: A tailored set of AI-driven transforms that optimizes Palo Alto Networks firewall log data by “numerifying” fields such as bytes, bytes_out, duration, aggregating events from the same source to different ports, and enriching data for improved security insights, SIEM compatibility, and cost reduction.
DNS Optimizer: A specialized set of AI-driven transforms that processes DNS log data by performing domain lookups, assigning ranks to unranked domains, filtering events, and removing temporary fields such as domain_rank to enhance security visibility, streamline SIEM integration, and lower storage costs.
Okta Optimizer: A targeted set of AI-driven transforms that optimizes Okta identity and access management log data by dropping successful SSO sessions and low-severity events, enriching data for better security monitoring, ensuring SIEM integration, and reducing operational costs.
Usage
All MultiTransform Optimizers are deployed within the targeted pipeline. Here is the deployment sequence:
Click the Add Transforms button or Plus (+) sign between any set of transforms or source/destinations
Click the Import button on the Add Transformation panel
Select Use an Observo Template radio button and click the Next button
On the Import Transform panel, select the targeted MultiTransform Optimizer such as Cisco ASA Optimizer
The targeted MultiTransform Optimizer appears within the pipeline as a set of transforms
Configuration Options
MultiTransform Optimizers are used to modify, enrich, or otherwise change event data as it flows through the pipeline. They offer operations such as adding or removing fields, filtering events, performing calculations, and masking sensitive data on specific source types such as Cisco ASA or Palo Alto firewalls. MultiTransform Optimizers are customizable upon deployment.
Cisco ASA Optimizer
This MultiTransform Optimizer consist of the following transforms:
Filter Events
Filter Cisco ASA events
Drops out non-critical CISCO ASA events (cisco.cisco_asa_event_id) such as 106011, 106013 or 106015.
Sample
Sample High volume events
Samples CISCO ASA events (cisco.cisco_asa_event_id) such as 106100 or 106023.
Palo Alto Optimizer
This MultiTransform Optimizer consist of the following transforms:
Numerify
Numerify Event Fields
Numerify targets a set of Palo Alto fields such as palo_alto.bytes, palo_alto.bytes_out and palo_alto.duration.
Reduce
Aggregate PaloAlto Events
Reduce Events from the same source to the same destination using different source ports.
DNS Optimizer
This MultiTransform Optimizer consist of the following transforms:
Lookup
Domain Lookup
This transforms performs a lookup of domain names and assigns rank to them. If the domain is not part of the lookup then rank is set as "unknown".
Filter Events
UnrankedDomainsFilter
This transforms allows only unranked domain events
Remove Fields
Remove domain_rank Field
This transform removes the domain_rank field that has been introduced while lookup
Okta Optimizer
This MultiTransform Optimizer consist of the following transforms:
Filter Events
DropSuccessSSOSession
This transform drops SSO events based on eventTypes.
Filter Events
DropLowSeverityEvents
This transform drops low severity events
Best Practices
Best practices for developing and implementing MultiTransform Optimizers, such as the Observo AI Cisco ASA, Palo Alto, DNS, and Okta Optimizers, involve a strategic approach to streamline data processing, enhance security, and ensure compatibility with external systems. Below are key best practices tailored to the described optimizers:
Source-Specific Transform Design: Tailor transforms to the unique characteristics of each data source, as demonstrated by the Cisco ASA Optimizer filtering non-critical events such as event IDs 106011, 106013, 106015 or the DNS Optimizer performing domain lookups to assign ranks to unranked domains.
AI-Driven Contextual Enrichment: Use AI-powered transforms, like the Sentiment Analyzer, to enrich data with actionable insights, such as ranking domains in the DNS Optimizer or “numerifying” fields in the Palo Alto Optimizer (e.g., bytes, bytes_out, duration) for enhanced analytics.
Targeted Data Reduction: Apply filters and sampling to reduce data volume while preserving critical information, as seen in the Cisco ASA Optimizer’s sampling of high-volume events such as 106100, 106023 or the Palo Alto Optimizer’s aggregation of events from the same source to different ports.
Standardized Output Formatting: Leverage serializers to format data into schemas compatible with external systems like SIEMs, ensuring seamless integration, as implied by the optimizers’ compatibility with platforms like Splunk or AWS Security Lake.
Dynamic Event Filtering: Implement precise filtering to eliminate irrelevant or low-priority data, such as the Okta Optimizer dropping successful SSO sessions and low-severity events to focus on high-risk incidents.
Scalable Pipeline Deployment: Deploy optimizers within the data pipeline using a streamlined process such as selecting the “Use an Observo Template” option, ensuring transforms like the DNS Optimizer’s removal of temporary fields such as domain_rank maintain pipeline efficiency.
Customizable Transform Configurations: Allow customization during deployment to adapt transforms to specific needs, such as modifying the Palo Alto Optimizer’s aggregation logic or adjusting the Okta Optimizer’s event filtering criteria.
Real-Time Anomaly Detection: Integrate AI-driven anomaly detection and sentiment scoring, as used across all optimizers, to prioritize alerts and reduce alert fatigue, such as flagging unranked DNS domains or high-risk Okta authentication events.
Cost and Storage Optimization: Use aggregation, sampling, and field removal such as DNS Optimizer’s removal of domain_rank to minimize storage and compute costs, achieving significant reductions (potentially over 50%, as per Observo’s claims).
Security and Compliance Focus: Ensure data processing occurs within the user’s environment, as emphasized by Observo AI, to meet compliance requirements, particularly for sensitive Okta identity logs or Cisco ASA firewall data.
These refined best practices ensure that MultiTransform Optimizers like the Cisco ASA, Palo Alto, DNS, and Okta Optimizers deliver efficient, secure, and scalable data processing tailored to diverse source types while maximizing value for security and observability workflows.
Resources
Here are some external reference resources that can provide valuable insights and context for understanding MultiTransform Optimizers, data processing, our data pipeline platforms, focusing on areas such as security, log management, and AI-driven data optimization:
Open Cybersecurity Schema Framework (OCSF)
Website: https://ocsf.io/
Description: OCSF is an open standard for structuring and normalizing security telemetry data, which aligns with Observo AI’s goal of seamless SIEM integration. It’s a great resource for understanding how optimizers like Cisco ASA or Okta ensure compatibility with external systems.
Relevance: Provides schema details for firewall logs such as Cisco ASA, Palo Alto, DNS, and identity data such as Okta, which are processed by Observo’s transforms.
AWS Security Lake Documentation
Description: AWS Security Lake is a centralized data lake for security telemetry, often integrated with platforms like Observo AI. Its documentation covers best practices for ingesting and optimizing logs from sources like firewalls and identity systems.
Relevance: Offers guidance on cost-effective storage and analytics, complementing the cost reduction goals of MultiTransform Optimizers.
Splunk Documentation – Data Ingestion and Parsing
Description: Splunk’s resources detail how to parse, filter, and enrich log data for security and observability, mirroring the functions of Observo’s parsers and serializers.
Relevance: Useful for understanding how Cisco ASA or Palo Alto logs are structured for SIEMs, as described in the MultiTransform Optimizers.
MITRE ATT&CK Framework
Website: https://attack.mitre.org/
Description: MITRE ATT&CK provides a knowledge base of adversary tactics and techniques, which can inform AI-driven anomaly detection in optimizers like the DNS or Okta Optimizer.
Relevance: Helps contextualize how Observo’s Sentiment Analyzer prioritizes alerts based on potential threats in firewall or identity logs.
Cloud Native Computing Foundation (CNCF) – Observability Resources
Website: https://www.cncf.io/projects/
Description: CNCF hosts projects like OpenTelemetry, which standardizes log, metric, and trace collection, offering best practices for observability pipelines.
Relevance: Aligns with Observo AI’s handling of diverse datasets (logs, metrics, traces) and provides insights into scalable pipeline design.
Gartner Research – Security Information and Event Management (SIEM)
Description: Gartner’s SIEM reports (often accessible via subscription or institutional access) discuss trends in log management and AI-driven analytics, relevant to Observo’s platform.
Relevance: Offers high-level insights into how MultiTransform Optimizers fit into modern SIEM workflows and cost optimization strategies.
These resources provide a mix of technical standards, practical guides, and strategic insights to deepen understanding of MultiTransform Optimizers and their role in security and observability. Always verify the latest updates on these sites, as they evolve with industry trends.
Last updated
Was this helpful?