AWS ALB Access Logs

Overview

AWS Application Load Balancer (ALB) access logs provide detailed information about incoming HTTP/HTTPS requests, including timestamps, client IP addresses, requested URLs, response codes, and user-agent details. These logs aid in monitoring traffic patterns, troubleshooting issues, and analyzing application performance within AWS infrastructure.

The volume of ALB Access Logs scales linearly with increase in network traffic to the ALB. As a result, it becomes expensive to store and process full fidelity of network logs. Observo helps optimize ALB Access logs allowing you to save costs in processing ALB logs while also increasing signal in existing logs.

Event Structure

Observo support parsing of ALB logs using the "AWS ALB Logs Parser" transform. This allows translation of to a much richer and structured format. The format of logs after processing by the above transforms is as follows:

{
  "actions_executed": "forward",
  "chosen_cert_arn": null,
  "classification": null,
  "classification_reason": null,
  "client_host": "192.168.10.11:2001",
  "domain_name": null,
  "elb": "test-alb/28nb1j581626",
  "elb_status_code": "200",
  "error_reason": null,
  "matched_rule_priority": "1",
  "received_bytes": 20,
  "redirect_url": null,
  "request_creation_time": "2023-11-10T10:12:32.371000Z",
  "request_method": "POST",
  "request_processing_time": 0,
  "request_protocol": "HTTP/1.1",
  "request_url": "http://www.request.com:80/",
  "response_processing_time": 0,
  "sent_bytes": 300,
  "ssl_cipher": null,
  "ssl_protocol": null,
  "target_group_arn": "arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/test-tg/271nmasb2374b",
  "target_host": null,
  "target_port_list": [],
  "target_processing_time": 0.02,
  "target_status_code": "200",
  "target_status_code_list": [],
  "timestamp": "2023-11-10T10:12:32.381000Z",
  "trace_id": "Root=1-18726361-12jkbn258972n1",
  "type": "http",
  "user_agent": "test-agent"
}

Aggregate Repetitive Logs

A large contribution of log volume usually stems from a small number of hosts. Further, hosts tend to send bursts of events in a short window of time. Using Observo's Reduce transform, we can aggregate repetitive logs that originate from the same source within a short period of time.

Create a Reduce transform in your pipeline with the following configuration:

Max Events: 10

Batch Flush Timeout (seconds): 30

Reduce Conditions:

Field Name: "target_status_code"
Regular Expression Condition: "200"

Group By:

actions_executed
elb
elb_status_code
matched_rule_priority
request_method
request_protocol
request_url
target_group_arn
type
user_agent

Reduction Methods:

received_bytes: Sum values
request_creation_time: Keep first value
request_processing_time: Sum values
response_processing_time: Sum values
sent_bytes: Sum values
target_processing_time: Sum values
timestamp: Keep last value
trace_id: Keep last value