AWS ALB Parser
The AWS ALB Parser in Observo AI is used to transform AWS Application Load Balancer (ALB) Access logs from raw text to a rich structured format.
Purpose
The purpose of an AWS ALB (Application Load Balancer) Parser is to extract, normalize, and structure log data from ALB access logs for easier analysis and ingestion into SIEMs or log management platforms. It helps convert raw log entries into readable and structured formats like JSON, ensuring compatibility with security and monitoring tools. By parsing and categorizing key details such as request timestamps, source IPs, response codes, and latencies, the parser enhances visibility, troubleshooting, and security analysis.
Usage
Select AWS ALB Parser transform. Add Name (required) and Description (optional).
General Configuration:
Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.
Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.
AWS ALB Parser: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.
Field Name: Field name on whose value the AWS ALB log parser will apply. Default: message. The supported ALB format is described here:
New Field Name: Specify the name of the destination field for storing the parsed data. If left empty, the parsed data will be written to the root level. Existing root level fields will be replaced with the new values if their field names match.
Preserve Original: Preserve the original field after parsing. If set to true, the raw event is stored at _ob.raw
Examples
Parse AWS ALB Logs
Scenario: Parse ALB logs and convert them into a more structured and enriched format.
Input
2024-02-27T12:34:56.123456Z app/my-load-balancer/123456789abcdef 192.168.1.1:56789 10.0.0.1:80 0.000 0.002 0.000 200 200 0 123 456 "GET https://example.com:443/index.html HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-target-group/abcdef123456 "Root=1-abcdef12-34567890abcdef1234567890" "-" "-" 0 2024-02-27T12:34:56.123456Z "forward" "-" "-"
2024-02-27T12:35:10.654321Z app/my-load-balancer/123456789abcdef 203.0.113.5:45678 10.0.0.2:443 0.001 0.005 0.000 404 404 0 321 654 "POST https://example.com:443/api/data HTTP/1.1" "curl/7.68.0" TLSv1.3 TLS_AES_256_GCM_SHA384 arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-target-group/abcdef123456 "Root=1-abcdef12-abcdef123456789012345678" "-" "-" 0 2024-02-27T12:35:10.654321Z "forward" "-" "-"
2024-02-27T12:36:20.789012Z app/my-load-balancer/123456789abcdef 198.51.100.10:34567 10.0.0.3:443 0.002 0.003 0.001 500 500 0 789 987 "PUT https://example.com:443/upload HTTP/1.1" "PostmanRuntime/7.26.8" TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384 arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-target-group/abcdef123456 "Root=1-abcdef12-0987654321abcdef12345678" "-" "-" 0 2024-02-27T12:36:20.789012Z "forward" "-" "-"Output
[
{
"timestamp": "2024-02-27T12:34:56.123456Z",
"load_balancer": "app/my-load-balancer/123456789abcdef",
"client_ip": "192.168.1.1",
"client_port": 56789,
"target_ip": "10.0.0.1",
"target_port": 80,
"request_processing_time": 0.000,
"target_processing_time": 0.002,
"response_processing_time": 0.000,
"elb_status_code": 200,
"target_status_code": 200,
"received_bytes": 123,
"sent_bytes": 456,
"request_method": "GET",
"request_url": "https://example.com:443/index.html",
"http_version": "HTTP/1.1",
"user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
"tls_version": "TLSv1.2",
"cipher_suite": "ECDHE-RSA-AES128-GCM-SHA256",
"target_group_arn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-target-group/abcdef123456",
"trace_id": "Root=1-abcdef12-34567890abcdef1234567890",
"actions_executed": "forward"
},
{
"timestamp": "2024-02-27T12:35:10.654321Z",
"load_balancer": "app/my-load-balancer/123456789abcdef",
"client_ip": "203.0.113.5",
"client_port": 45678,
"target_ip": "10.0.0.2",
"target_port": 443,
"request_processing_time": 0.001,
"target_processing_time": 0.005,
"response_processing_time": 0.000,
"elb_status_code": 404,
"target_status_code": 404,
"received_bytes": 321,
"sent_bytes": 654,
"request_method": "POST",
"request_url": "https://example.com:443/api/data",
"http_version": "HTTP/1.1",
"user_agent": "curl/7.68.0",
"tls_version": "TLSv1.3",
"cipher_suite": "TLS_AES_256_GCM_SHA384",
"target_group_arn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-target-group/abcdef123456",
"trace_id": "Root=1-abcdef12-abcdef123456789012345678",
"actions_executed": "forward"
},
{
"timestamp": "2024-02-27T12:36:20.789012Z",
"load_balancer": "app/my-load-balancer/123456789abcdef",
"client_ip": "198.51.100.10",
"client_port": 34567,
"target_ip": "10.0.0.3",
"target_port": 443,
"request_processing_time": 0.002,
"target_processing_time": 0.003,
"response_processing_time": 0.001,
"elb_status_code": 500,
"target_status_code": 500,
"received_bytes": 789,
"sent_bytes": 987,
"request_method": "PUT",
"request_url": "https://example.com:443/upload",
"http_version": "HTTP/1.1",
"user_agent": "PostmanRuntime/7.26.8",
"tls_version": "TLSv1.2",
"cipher_suite": "ECDHE-RSA-AES256-GCM-SHA384",
"target_group_arn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-target-group/abcdef123456",
"trace_id": "Root=1-abcdef12-0987654321abcdef12345678",
"actions_executed": "forward"
}
]Results: The key benefits of parsing AWS ALB logs include improved readability, easier querying, enhanced security monitoring, and faster troubleshooting through structured and enriched log data.
Best Practices for Parsing AWS ALB Logs
Use a Structured Format – Convert raw logs into JSON for easier querying and analysis in SIEM and log management tools.
Normalize Timestamps – Store timestamps in UTC (ISO 8601 format) to ensure consistency across different systems.
Extract Key Fields – Parse and retain critical data such as client IP, request URL, response codes, latency, and user agent for security and performance monitoring.
Remove Unnecessary Data – Filter out redundant or irrelevant fields to reduce storage costs and improve processing efficiency.
Ensure Proper Field Mapping – Align parsed fields with Splunk CIM or other standard schemas for better log correlation and analysis.
Monitor for Parsing Errors – Regularly validate logs for inconsistencies, missing values, or incorrect field extraction to maintain data accuracy.
Preserve Original Log Data – Keep raw logs accessible for forensic investigations and troubleshooting when needed.
Implement Log Rotation & Retention Policies – Manage log storage effectively by setting appropriate retention policies based on compliance and security needs.
Use Log Enrichment – Augment parsed logs with geolocation, threat intelligence, or additional metadata for enhanced security insights.
Automate Log Ingestion & Parsing – Utilize Observo AI Pipelines to streamline log parsing and integration.
By following these best practices, organizations can enhance visibility, improve security monitoring, and optimize AWS ALB log management
Related Functions
Syslog Parser: Parse Syslog event into structured JSON.
CEF Parser: Extracts and normalizes fields from CEF-formatted logs, enabling efficient search, correlation, and analysis in SIEM systems.
Last updated
Was this helpful?