Encode Fields

Encode field values within log events. It offers a variety of encoding methods including URL Encode, Zlib, Zstd, Gzip, JSON, Base 16, and Base 64. By enabling this processor, you can transform field values using the specified encoding types.

Purpose

Encoding log field values ensures that sensitive information, such as user data or system details, is protected during storage and transmission. It helps maintain data integrity by standardizing the format, making it easier to parse and analyze across different systems. Additionally, encoding prevents malicious injections and tampering, improving the security and reliability of the logging process.

Usage

Select Encode Fields transform. Add Name (required) and Description (optional).

General Configuration:

• Bypass Transform: Defaults to disabled. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.

• Add Filter Conditions: Defaults to disabled. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

EncodeFields: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.

Fields to Encode Rules: Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs:

• Field to Encode: The name of the field whose value is to be encoded.

• Encoding Type: The type of encoding to apply to the field.

The Encoding Type options provided are:

• URL Encode: Encodes the field value using URL encoding, also known as percent encoding. It replaces special characters with their hexadecimal representation.

• Zlib: Compresses the field value using the Zlib compression algorithm. The resulting value is then base64 encoded.

• Zstd: Compresses the field value using the Zstandard compression algorithm. The resulting value is then base64 encoded.

• Gzip: Compresses the field value using the gzip compression algorithm. The resulting value is then base64 encoded.

• JSON: Encodes the field value as a JSON string.

• Base 16: Encodes the field value using base 16 (hexadecimal) representation.

• Base 64: Encodes the field value using base 64 encoding.

Examples

Field to Encode	Field Content	Encoding Type	Output
hostname	make.com	Base64	bWFrZS5jb20=
msgid	ID166	Zlib	eJzzdDE0MwMAA7cBKw==
severity	err	Zstd	KLv/QBYGQAAZXJy

Best Practices

1. Identify and Prioritize Sensitive Data

   • Focus on encoding sensitive or personally identifiable information (PII) such as emails, IP addresses, credit card numbers, or Social Security numbers.

   • Avoid over-encoding non-sensitive fields to maintain data usability.

2. Choose the Right Encoding Method

   • Base64: For general-purpose encoding of text or binary data.

   • URI Encoding: For URLs or query parameters.

   • GZIP: For compressing large fields to save storage and bandwidth.

3. Combine Encoding with Masking

   • Use masking alongside encoding to ensure sensitive data cannot be easily reversed.

   • Example: Hash a password before Base64 encoding it

4. Preserve Data Usability

   • Use reversible encoding such as Base64 for non-sensitive data to allow debugging or analysis.

   • Add metadata (e.g., encoded: true) to indicate which fields are encoded.

   • Document encoding rules and provide decoding keys when necessary.

5. Standardize Encoding Across Logs

   • Use consistent encoding methods for the same types of data such as Base64 for emails.

   • Document and share encoding standards with your team to ensure consistency.

6. Test and Monitor Encoding

   • Test encoding in a staging environment to ensure compatibility with downstream systems such as SIEMs or analytics tools.

   • Monitor performance to avoid bottlenecks caused by encoding large volumes of data.

Related Functions

• Sensitive Data Mask: Obfuscate sensitive information in your data streams.

• Decode Fields: Decode field values in log events.

• Filter Data: Apply conditions to filter data before or after adding fields.

Additional Resources

• What are some common pitfalls and best practices of data encoding and transformation?