AWS VPC Flow Logs Parser

The AWS VPC Flow Logs Parser in Observo AI parses AWS VPC Flow Logs to JSON, extracting and structuring the logs for streamlined analysis.

Purpose

The AWS VPC Flow Logs Parser extracts, normalizes, and structures network traffic logs from AWS Virtual Private Cloud (VPC) environments. It converts raw flow log data into a readable format, enabling security and operations teams to analyze network activity, detect anomalies, and troubleshoot connectivity issues efficiently. By parsing key details such as source/destination IPs, ports, protocols, and traffic statuses, the parser enhances visibility into network traffic and supports compliance, threat detection, and performance monitoring.

Usage

Select AWS VPC Flow Logs Parser transform. Add Name (required) and Description (optional).

General Configuration:

• Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.

• Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

Parser Config:

Format Strings to use for parsing VPC flow logs: Click Add button to add a Format String. Add as many format strings as required. It is the same string defined in the configuration of the VPC Flow Logs. If multiple format strings are specified, the first one that matches will be used.

Examples:

• version vpc_id region subnet_id instance_id interface_id account_id type srcaddr dstaddr srcport dstport pkt_srcaddr pkt_dstaddr protocol bytes packets start end action tcp_flags log_status

• ${version} ${vpc-id} ${region} ${subnet-id} ${instance-id} ${interface-id} ${account-id} ${type} ${srcaddr} ${dstaddr} ${srcport} ${dstport} ${pkt-srcaddr} ${pkt-dstaddr} ${protocol} ${bytes} ${packets} ${start} ${end} ${action} ${tcp-flags} ${log-status}

Field To Parse: Field in the log that contains the raw VPC Flow Log. This field will be parsed using the format strings specified above. Default: message.

Drop Header Events: If true, the processor will drop events that are headers. If false, the processor will not drop header events. Header events are events that do not contain any data and are only used to describe the format of the log.

Example

Parse AWS VPC Flow Logs

Scenario: Parse AWS VPC Flow logs and convert them into a more structured and enriched format.

Format Strings to use for parsing logs	Field To Parse	Drop Header Events
account_id action bytes dstaddr dstport end instance_id interface_id log_status packets pkt_dstaddr pkt_srcaddr protocol srcaddr srcport start subnet_id tcp_flags timestamp type version vpc_id	message	False

Input

{
  "message": "4 130917795281 eni-0bbb5036baadb8xxx xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx 31921 443 6 23 6132 1674689202 1674689233 ACCEPT OK"
}

Output

{
  "_ob": {
    "error": null,
    "schema_id": "2e7a7ce82578732ff1x1x2x3x4x5x6f7b",
    "source": "100000000000000073"
  },
  "account_id": 130917795281,
  "action": "ACCEPT",
  "bytes": 6132,
  "dstaddr": "xxx.xxx.xxx.xxx",
  "dstport": 443,
  "end": 1674689233,
  "instance_id": "i-0b26442fc32d39xxx",
  "interface_id": "eni-0bbb5036baadb8xxx",
  "log_status": "OK",
  "packets": 23,
  "pkt_dstaddr": "xxx.xxx.xxx.xxx",
  "pkt_srcaddr": "xxx.xxx.xxx.xxx",
  "protocol": 6,
  "srcaddr": "xxx.xxx.xxx.xxx",
  "srcport": 31921,
  "start": 1674689202,
  "subnet_id": "subnet-0f9a25c96afaf7xxx",
  "tcp_flags": 3,
  "timestamp": "2023-08-08T17:38:59Z",
  "type": "IPv4",
  "version": 4,
  "vpc_id": "vpc-0fb815a01a69a8xxx"
}

Results: The key benefits of parsing AWS VPC Flow Logs include improved readability, enhanced security analysis, easier querying, better compliance tracking, and optimized troubleshooting of network traffic.

Best Practices for Parsing AWS VPC Flow Logs

1. Use a Structured Format – Convert raw logs into JSON for easier analysis and integration with SIEM tools.

2. Normalize Timestamps – Ensure logs use UTC (ISO 8601 format) for consistency across different systems.

3. Extract Key Fields – Focus on critical details such as source/destination IPs, ports, protocols, actions, and bytes transferred for security and troubleshooting.

4. Filter Unnecessary Logs – Reduce storage and processing costs by filtering irrelevant or redundant log entries.

5. Map Fields to a Standard Schema – Align parsed data with CIM (Common Information Model) or security frameworks for better correlation.

6. Monitor for Anomalies – Set up alerts for unusual patterns, high traffic spikes, excessive rejected connections, or unexpected IPs.

7. Preserve Raw Logs – Keep unmodified logs for forensic investigations and compliance audits when needed.

8. Optimize Log Storage & Retention – Implement efficient log rotation and retention policies based on security and regulatory requirements.

9. Use Enrichment – Augment logs with geolocation, threat intelligence, or AWS metadata to provide additional security context.

10. Automate Parsing & Ingestion – Use Observo AI Pipelines to streamline log processing and improve data accuracy.

Following these best practices enhances security monitoring, compliance, and operational efficiency when working with AWS VPC Flow Logs.

Related Functions

• Syslog Parser: Parse Syslog event into structured JSON.

• CEF Parser: Extracts and normalizes fields from CEF-formatted logs, enabling efficient search, correlation, and analysis in SIEM systems.