Cloudtrail
The Cloudtrail Optimizer in Observo AI allows users to process AWS Cloudtrail events.
Purpose
A transform tailormade for Cloudtrail. It includes options to:
Drop events based on eventnames by matching it with supplied regexes. Usually it helps to reduce volume to drop eventnames with List, Get and such by using a regex like List*|Get*|Describe*
Applying sensitive data masking in Observo AI automatically redacts credentials and identifiers from logs to enhance security, support compliance such as SOC 2, PCI-DSS, HIPAA, and optimize operations by safely reducing log volume and protecting downstream systems.
Specify exact keys to remove from each CloudTrail event, helping reduce payload size and eliminate unnecessary or sensitive fields before downstream processing.
Perform aggregations on incoming Cloudtrail events by using more fine-grained aggregation policies.
Usage
Select Cloudtrail Optimizer transform. Add Name (required) and Description (optional).
General Configuration:
Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.
Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.
Cloudtrail Optimizer:
Filter By Event Names (pulldown):
Enabled (True): Enable or disable this transform.
Regex for Eventnames: Input regex that will be used to filter out events based on event names. Click the Add button to add a new field. Click on Trash Can Icon to delete entry.
Examples:List*
Get*
Describe*
Mask Sensitive Data (pulldown):
Enabled (True): Enable or disable this transform.
Mask Access Keys (True): Masks Access Keys
Mask Assume RoleIds (True): Masks Assume RoleIds
Event Serializer (pulldown):
Enabled (True): Enable or disable this transform.
Keys to remove: Specify keys that will be removed from each Cloudtrail event. Click the Add button to add a new field. Click on Trash Can Icon to delete entry.
Examples:additionalEventData
requestParameters
tlsDetails
userIdentity.accessKeyId
Aggregation (pulldown):
Enabled (True): Enable or disable this transform.
Field names to Aggregate By: A comma separate list of columns to group by and merge. Add button to add a new field. Click on Trash Can Icon to delete entry.
Examples:eventName
sourceIPAddress
requestParameters
responseElements
eventCategory
Max Events: The maximum number of events to group together. Default: 100.
Flush Time(seconds): The maximum amount of time in seconds to wait before flushing events to Destination. Defaults: 30.
Aggregation Methods: Set of event fields to evaluate and add/set. Click Add button to add new field as a key-value pair, with the following inputs:
Field Name: The name of the field whose value is being aggregated.
Aggregation Method: The method used to aggregate the values of the field. For example, if the field is an integer, you can sum the values, or keep the maximum value. If the field is a string, you can concatenate the values, or keep the shortest or longest value.
Here are the possible methods:
Append value to array
Concatenate string separated by space
Concatenate string separated by newline
Concatenate string without any separator
Keep first value
Keep last value
Create flattened array of unique values
Keep maximum value
Keep minimum value
Sum values
Keep shortest array
Keep longest array
Examples
Let’s use the following AWS Cloudtrail Data Snippet to separately validate the Filter By Event Names, Mask Sensitive Data, and Aggregation.
Filter By Event Names
Scenario: Drop events based on event names by matching it with supplied regexes. Here we reduce the volume by dropping event names with List, Get using a regex - List*|Get*|Describe*
List*
Get*
Describe*
Results: Events containing List, Get or Describe are dropped.
Mask Sensitive Data
Scenario: An organization uses Observo AI to ingest AWS CloudTrail logs and applies sensitive data masking to redact credentials and role identifiers before routing logs to analytics and storage platforms, ensuring security and compliance.
Results:
Enhanced Security
Access Keys and Assume Role IDs are stripped before they ever hit downstream systems
Reduces blast radius if a log leak or misconfiguration occurs
Compliance Alignment
Ensures PII and credential data do not leave your trusted zones in clear text.
Enables smooth audits for:
SOC 2: Protects sensitive data exposure
PCI-DSS: Prevents PAN/credential storage in logs
ISO 27001 / HIPAA: Ensures data minimization
Operational Benefits:
Splunk usage drops by ~20–30% due to reduced log payload size.
No need to manually redact sensitive fields in dashboards or alerts.
Engineers and analysts can still see relevant context (eventName, roleSessionName, region, etc.) without the risk.
Event Serializer
Scenario: Specify exact keys to remove from each CloudTrail event, helping reduce payload size and eliminate unnecessary or sensitive fields before downstream processing.
additionalEventData
responseElements
Results: Only the awsRegion log entry parts are kept while the additionalEventData and the responseElements log entry parts are discarded.
Aggregation
Scenario: Perform aggregations on incoming Cloudtrail events by using more fine-grained aggregation policies. Keep Max Events to default:100 and Flush Time(seconds) to default: 30.
eventName
sourceIPAddress
requestParameters
responseElements
responseElements
eventCategory
Results: Cloudtral logs are aggregated to a set of manageable patterns
Cloudtrail Optimizer Best Practices
Below is an overview of best practices for optimizing your CloudTrail logs with Observo AI’s Optimizer using three key techniques:
Filter Event Names
Prioritize Critical Events:
Configure your optimizer to only pass through CloudTrail events that are actionable or critical for security, compliance, and operations.
Exclude routine or “noisy” event names (such as benign read operations that do not add analytical values.
Tailor to Your Use Case:
Use domain knowledge to create allow/deny lists for event names based on your monitoring objectives.
Dynamically adjust filters as your environment and threat landscape evolve.
Event Serializer (Drop Unused Fields)
Identify Redundant Data:
Analyze the full CloudTrail JSON payload and determine which keys (fields) are not used for alerting or analysis.
Remove keys that are constant across events or that provide little variance.
Reduce Data Volume:
Dropping unnecessary keys at ingestion minimizes storage and improves downstream query performance.
Ensure that only the metadata critical for your analysis, such as eventName, eventTime, userIdentity, and resource identifiers, is retained.
Mask Sensitive Data
Mask Access Keys:
Access keys can be used to authenticate and authorize AWS API calls. Exposure of these keys could lead to unauthorized access.
Best Practice: Always mask or redact access keys (
accessKeyId,secretAccessKey) before sending logs to downstream systems.
Mask Assume Role IDs:
AssumeRole events include role ARNs and session names that may reveal internal IAM structure or identities.
Best Practice: Mask
roleArn,sessionContext, andprincipalIdfields to avoid leaking role usage patterns or sensitive identity details.
Audit Your Masking Settings:
Periodically review masking settings to ensure they meet changing security requirements and compliance obligations such as GDPR, HIPAA, FedRAMP.
Aggregation
Smart Summarization:
Group similar events by key identifiers such as eventName, user or resource IDs over a defined time window.
Use machine learning–powered aggregation to summarize high-frequency events into a single representative record that retains counts, time ranges, and key metrics.
Reduce Noise & Improve Efficiency:
Aggregation not only reduces the log volume by merging repetitive events but also speeds up analysis by transforming detailed logs into digestible summaries.
Maintain the ability to drill down into aggregated data when detailed analysis is required.
Additional Recommendations:
Automate and Iterate
Set up dynamic pipelines so that filtering, key removal, and aggregation run automatically without manual intervention.
Regularly review and fine-tune your rules to ensure they stay aligned with evolving business needs and threat scenarios.
Maintain Analytical Integrity
While reducing data volume, verify that essential context isn’t lost—ensure that alerts, forensics, and compliance reporting remain robust.
By applying these best practices, Observo AI’s CloudTrail Optimizer helps streamline your log data—ensuring that you retain only the actionable events with essential context, reduce storage costs, and improve query performance.
Related Functions
GCP Flow Log Optimizer: Perform various aggregations, including smart summarization, on GCP Flow Logs data.
AWS VPC Flow Logs: Perform various optimizations, including smart summarization, on AWS VPC Flow Logs data.
Last updated
Was this helpful?