Reducing Data

Event Filtering

Use Case: Filtering and refining event data to exclude unwanted events or specific log levels (e.g., Info/Debug/Trace).

Example: Suppose you want to filter out debug and trace events to reduce noise in your log data.

Transforms:

• Regular Expression (Regex) Filtering: Define regex patterns to match and filter logs.

• Drop Events: Remove specific log events based on predefined criteria.

• Sampling: Sample a subset of events for analysis.

Field Filtering

Use Case: Selectively retaining or removing specific fields within event entries.

Example: You might want to keep only essential fields like "timestamp," "log message," and "error code" while discarding other fields in the event.

Transforms:

• Remove Fields: Condition based removal of unwanted fields from events.

Throttle Events

Use Case: Control the rate at which events or log entries are processed or forwarded to prevent system overload or excessive resource consumption.

Example:

1. Rate Limiting: Enforce rate limits to prevent an overwhelming influx of events, ensuring the system can handle incoming log data without degradation in performance.

2. Preventing Resource Exhaustion: Throttling can be used to avoid overloading downstream systems, such as databases or monitoring tools, to prevent resource exhaustion and maintain system stability.

3. Mitigating DDoS Attacks: In the case of Distributed Denial of Service (DDoS) attacks, throttling can help mitigate the impact by limiting the number of incoming requests, reducing the strain on the targeted resources.

4. Controlling Costs: Throttling can help control costs associated with log processing and storage by limiting the volume of data ingested into your logging system.

Transforms:

• Throttle: Define condition based throttling of events.

Duplicate Events Removal

Use Case: Eliminating duplicate events or redundant fields to reduce data redundancy.

Example: Removing duplicate events that may occur due to multiple logging instances.

Transforms:

• Deduplication: Identify and remove duplicate log events or fields.

Aggregate events

Use Case: Aggregating events that share common field values for better data organization.

Example: Grouping logs by a specific field, like "customer ID," to analyze customer-specific logs together.

Transforms:

• Reduce: Summarize logs by defining a custom identifier and rules for aggregation of fields.

Summarize Exceptions Across Events

Use Case: Summarizing and consolidating exception-related events for easier troubleshooting.

Transforms:

• Exception Summarization: Summarize exceptions by grouping and summarizing related log events.

Consolidate events based on Trace ID

Use Case: Summarizing events related to a single trace ID or transaction for comprehensive analysis.

Transforms:

• Trace Summarization: Summarize logs by trace ID to eliminate metadata redundancy.

Pattern based reduction

Use Case: Use patterns to identify events that are major contributors to log volume.

Transforms:

• Drop Events: Remove specific log events by defining a matching regular expression.