Elastic Audit Logs

Overview

Elasticsearch provides audit logging feature to record events related to the security of the Elasticsearch cluster. These events can include authentication and authorization activities, access to indices, and other security-relevant operations. Elasticsearch audit logging is a essential element for security monitoring and compliance. The enduring retention of these logs is beneficial for security analysis, incident response, continuous improvement.

Log Formats

When auditing security events, a single client request can produce numerous audit events spanning multiple cluster nodes. These logs are created in JSON format, and the following example represents a type that constitutes the majority of the volume.

{
  "type":"audit",
  "timestamp":"2023-12-08T05:06:56,473+0000",
  "cluster.uuid":"kXrjmQofTJWWwA_XXXX-MQ",
  "node.id":"J_-jlr-3Qo-XXXXXXXXX",
  "event.type":"transport",
  "event.action":"access_granted",
  "authentication.type":"API_KEY",
  "user.name":"elastic/fleet-server",
  "user.realm":"_service_account",
  "user.roles":[
    "elastic/fleet-server"
  ],
  "apikey.id":"y4UDQ4wB31InbpVXXXX",
  "apikey.name":"00bcf078-XXXXX-4946-a680-XXXXXXXX:default",
  "origin.type":"rest",
  "origin.address":"10.42.13.244:43252",
  "request.id":"zOeZEqYyQtW_GPIZG4yFZA",
  "action":"indices:data/write/bulk[s]",
  "request.name":"BulkShardRequest",
  "indices":[
    "metrics-kubernetes.node-default"
  ],
  "x_forwarded_for":"118.179.32.193"
}

Additional information about each type of audit log event can be found here in Elastic's official documentation.

Optimization Strategies

In our examination of a typical Elasticsearch cluster, over 99% of Elastic audit events indicate that the action is "access_granted". Therefore, our optimization efforts will be concentrated on this specific type of audit event.

We have outlined two optimization strategies below with varying degrees of data optimization. You can select the one that best aligns with your use case.

Aggregate Bulk Shard Requests

The majority of event logs we've observed are BulkShard Requests, which are generated when Elasticsearch writes bulk data across different indices. Additionally, the logging includes information about the replication process for BulkShardRequest, documenting the writes to its replicas. So we can aggregate logs with same "request.id", "request.name" and 'indices' but different action. This approach is expected to yield a 25%-30% optimization.

Reduce Conditions:

Field Name: "request.name"
Regular Expression Condition: BulkShardRequest

Field Name: "event.action"
Regular Expression Condition: access_granted

Group By:

"cluster.uuid"
"node.id"
"request.id"
"indices"

Reduction Methods:

action: Flatted array of unique values
request_id: Keep last value

Elastic Audit Logs optimization pipeline

Aggregate All Requests

This approach extends beyond just BulkShardRequests to encompass all types of actions. While it offers optimization gains of around 60%-70%, it comes with the trade-off of potential data order loss during aggregation.

Reduce Conditions:

Field Name: "event.action"
Regular Expression Condition: access_granted

Group By:

"cluster.uuid"
"node.id"
"request.id"
"request.name"
"indices"

Reduction Methods:

action: Flatted array of unique values
request_id: Keep last value

Elastic Audit Logs optimization pipeline