Remove Fields
The Remove Fields function allows you to remove specific fields from your data during the transformation process. This function is useful for reducing clutter, improving performance, and ensuring compliance with data privacy policies.
Purpose
The Remove Fields function is typically used in transformation pipelines to:
Remove unnecessary or redundant fields.
Exclude sensitive or private data before storage or processing.
Streamline data by eliminating unused fields.
Usage
Select Remove Fields transform. Add Name (required) and Description (optional).
General Configuration:
Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.
Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.
Remove Fields: Delete Condition Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs:
Field Name: Enter field name used in the filter condition. Click in the associated text box to get field choices or type field name directly.
Regular Expression Condition: Enter a field condition. If specified, only keys whose values match the regular expression will be deleted. Only applicable to string fields.
Examples
Remove Specific Fields
Scenario: Remove the fields from all events:
palo_alto.future_use1
palo_alto.version
palo_alto.future_use3
Delete Conditions Rules
Rule 1
palo_alto.future_use1
n/a
Rule 2
palo_alto.version
n/a
Rule 3
palo_alto.future_use3
n/a
Results: The the palo_alto.future_use1, palo_alto.version and palo_alto.future_use3 fields are removed from all events.
Remove Field Conditionally
Scenario: Remove the palo_alto.start_time, palo_alto.receive_time, and palo_alto.version fields only if the environment field equals production.
Delete Conditions Rules
Enabled
AND
environment
equals
production
Rule 1
palo_alto.start_time
n/a
Rule 2
palo_alto.receive_time
n/a
Rule 3
palo_alto.version
n/a
Results: The palo_alto.start_time, palo_alto.receive_time, and palo_alto.version fields are removed if the condition label environment field equals the production.
Remove Nested Field
Scenario: Remove a nested field palo_alto from all test environment events.
Delete Conditions Rules
Enabled
AND
environment
equals
test
Rule 1
palo_alto
n/a
|
{
"appname":"pan",
"facility":"lpr",
"hostname":"fw_syslog", “environment”:”test”,
"palo_alto":{
"action":"allow",
"action_flags":"0x0",
"action_source":"from-policy",
"app":"incomplete",
"bytes":"9259",
"bytes_in":"0",
"bytes_out":"9259", … "version":"0",
"vsys":"vsys1",
"vsys_name":""
},
"severity":"alert",
"source_ip":"192.168.3.48",
"timestamp":"2025-02-20T16:41:53.851Z"
}|
{
"appname":"pan",
"facility":"lpr",
"hostname":"fw_syslog", “environment”,
"severity":"alert",
"source_ip":"192.168.3.48",
"timestamp":"2025-02-20T16:41:53.851Z"
}| | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
Results: The nested field palo_alto is removed from all test environment events.
Best Practices
Identify Unnecessary Fields: Regularly review your data to identify fields that are no longer needed.
Protect Sensitive Data: Use this function to remove sensitive or private information before storing or processing data.
Use Conditions Wisely: Apply conditions to remove fields only when necessary, avoiding unintended data loss.
Test Configurations: Test your remove_fields configurations in a development environment before deploying to production.
Monitor Performance: Removing fields can improve processing performance, but ensure critical data is not accidentally removed.
Related Functions
Add Fields: Add new fields to your data.
Rename Fields: Rename fields to standardize naming conventions.
Filter Event: Apply conditions to filter data before or after removing fields.
Additional Resources
Last updated
Was this helpful?