Throttle
The Throttle function allows you to limit the rate of events passing through your Observo AI pipeline. This is useful for controlling the volume of data sent to downstream systems, preventing overloading, or adhering to rate limits imposed by external services.
Purpose
The Throttle function is typically used in scenarios where you want to:
Prevent downstream systems from being overwhelmed by high event volumes.
Enforce rate limits for APIs or external services.
Smooth out spikes in event traffic to ensure consistent processing.
Use this with caution.
Usage
Select Throttle transform. Add Name (required) and Description (optional).
General Configuration:
Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.
Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.
Throttle: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.
Threshold: The maximum number of events allowed for a given time window. Defaults to 100.
Window in Seconds: The duration of time, measured in seconds, during which the defined threshold is enforced. Defaults to 1.
Fields to Exclude Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs (optional):
Field Name: Field name on which the regex expression will apply. Events that match this regex pattern of the field value will be excluded from the throttle mechanism. If multiple fields are specified, they will be ANDed.
Match Regex: Regex expression used to identify the field value on which the exclusion criteria will be applied. Events that match this regex pattern for the specified field will be excluded from the throttle mechanism.
Examples
Basic Throttling
Scenario: Limit events to 500 per second, with a burst capacity of 50.
Throttle
Toggled on
500
50
Fields to Exclude
[Empty]
[Empty]
Outcome: Event streaming is limited to 500 per seconds with burst capacity of 50.*
Throttling by Field
Scenario: Limit events to 100 per second for each unique user_id, with a burst capacity of 10.
Throttle
Toggled on
100
10
Fields to Exclude
user_id
.*
Outcome: Events with user_id are excluded from the throttling.
Use Cases
Rate Limiting: Enforce rate limits to prevent an overwhelming influx of events, ensuring the system can handle incoming log data without degradation in performance.
Preventing Resource Exhaustion: Throttling can be used to avoid overloading downstream systems, such as databases or monitoring tools, to prevent resource exhaustion and maintain system stability.
Mitigating DDoS Attacks: In the case of Distributed Denial of Service (DDoS) attacks, throttling can help mitigate the impact by limiting the number of incoming requests, reducing the strain on the targeted resources.
Controlling Costs: Throttling can help control costs associated with log processing and storage by limiting the volume of data ingested into your logging system.
Related Functions
Sample: Reduce the volume of events by setting sampling rate.
Filter Event: Apply conditions to filter data before or after removing fields
Additional Resources
Last updated
Was this helpful?