Lookup

The Lookup function enriches your data by matching fields in your events with values in a lookup table. This is useful for adding context, translating codes, or categorizing data.

Purpose

Use the Lookup function when you need to enrich events with additional information from a predefined dataset. Common use cases include:

• Translating IDs to human-readable names.

• Adding geographic or demographic information.

• Categorizing data based on specific criteria.

Usage

Select Lookup transform. Add Name (required) and Description (optional).

General Configuration:

• Bypass Transform: Defaults to disabled. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.

• Add Filter Conditions: Defaults to disabled. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

Lookup:

• Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.

• Filter Conditions: Defaults to empty. When set, allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

• Lookup CSV File: The path to the lookup file containing the lookup data. Do not enter the full path. The file must be uploaded to the site.

• Match Parameters Rules: Specify the mapping between the event fields and the lookup file columns. If multiple, all must match. Set of event fields to evaluate and add/set. First field entry (1 rule) key-value pair added by default. Click Add button to add new field as a key-value pair, with the following inputs:

  • Lookup Column Name: Add Lookup Column Name.

  • Field to Mask: Add Field to Mask.

• Output Fields Rules: Define the fields to output based on the lookup. If a field name exists in the event, it will be overwritten with the lookup value. Click Add button to add new field as a key-value pair, with the following inputs:

  • Field Name: Add Field Name.

  • Lookup Column Name: Add Column Name.

  • Default Value: Add Default Value.

Examples

Add GeoIP Information

Scenario: Add GeoIP information to log entry.

Examples require that the following criteria are meet:

LookupCriteria

The Lookup CSV File entry field is specified as geoip.csv
The geoip.csv file was uploaded to the site under Settings→Files

Lookup Configurations

Filter Conditions:
  field: event_type
  operator: equals
  value: web_access

Lookup CSV File: geoip.csv

Match Parameters:
  - Event Field Name: client_ip
    Lookup Column Name: ip_start

Output Fields:
  - New Field Name: geo_country
    Lookup Column Name: country
    Default Value: Unknown

  - New Field Name: geo_city
    Lookup Column Name: city
    Default Value: Unknown

Lookup CSV File geoip.csv content:

ip_start,ip_end,country,city
1.0.0.0,1.0.0.255,Australia,Sydney
1.0.1.0,1.0.3.255,China,Nanjing
...

Input

{
  "event_type": "web_access",
  "client_ip": "1.0.2.100",
  "url": "/index.html",
  "status": 200
}

Output

{
  "event_type": "web_access",
  "client_ip": "1.0.2.100",
  "url": "/index.html",
  "status": 200,
  "geo_country": "China",
  "geo_city": "Nanjing"
}

Results: The GeoIP information for city and country are added to the log entry.

Behavior

• Key Field: If the key field in the event does not match any entry in the lookup table, the event remains unchanged.

• Output Fields: If an output field is not found in the lookup table, it is omitted from the event.

• Lookup Table: The lookup table can be updated dynamically, and changes will be reflected in the enriched events.

Limitations

• The size of the lookup table can impact performance. Large tables may require optimization.

• Lookup tables must be kept up-to-date to ensure accurate enrichment.

Related Functions

• Filter Event: Apply conditions to filter data before or after removing fields.

• Aggregate Metrics: Aggregate multiple metrics into a single metric based on a set of conditions.

Additional Resources

• Observo AI Lookup File Management