Cloudwatch Subscription Log

The CloudWatch Subscription Log Parser in Observo AI allows users to parse Cloudwatch subscription log events into structured JSON.

Purpose

The CloudWatch Subscription Log Parser is designed to reverse the consolidation of events collected through CloudWatch Subscription. It reverts these merged events back to their original format, ensuring that you can easily analyze and process them in a more structured and meaningful manner. This transform outputs a list of structured events from the single Cloudwatch Subscription Log event.

Usage

Select CloudWatch Subscription Log Parser transform. Add Name (required) and Description (optional).

General Configuration:

Bypass Transform: Defaults to disable. When enabled, this transform will be bypassed entirely, allowing the event to pass through without any modifications.
Add Filter Conditions: Defaults to disable. When enabled, it allows events to filter through conditions. Only events that meet the true condition will be processed; all others will bypass this transform. Based on AND/OR conditions, "+Rule" or "+Group" buttons.

CloudWatch Subscription Log Parser: Enabled: Defaults to enabled, meaning it does evaluate all events. Toggle Enabled off to prevent event processing to feed data to the downstream Transforms.

Field Name: Field name to parse. This field contains the raw events. Default: message.

Example

Explode/Replace Original Field Content

Scenario: This configuration enables the Cloudwatch Subscription Parser transformation, and it will parse the "message" field in log data.

Cloudwatch Subscription Log Parser

Field Name

message

Input

{
  "host": "someone-mac.local",
  "logEvents": [
    {
      "id": "**********448",
      "message": "{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"6134a20d-7699-4311-bfe2-8eb8185a5dec\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/eks-certificates-controller\",\"verb\":\"update\",\"user\":{\"username\":\"eks:certificate-controller\",\"groups\":[\"system:authenticated\"]},\"sourceIPs\":[\"172.16.111.18\"],\"userAgent\":\"eks-certificates-controller/v0.0.0 (linux/amd64) kubernetes/$Format\",\"objectRef\":{\"resource\":\"leases\",\"namespace\":\"kube-system\",\"name\":\"eks-certificates-controller\",\"uid\":\"********\",\"apiGroup\":\"coordination.k8s.io\",\"apiVersion\":\"v1\",\"resourceVersion\":\"60536531\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2023-09-21T20:43:41.300753Z\",\"stageTimestamp\":\"2023-09-21T20:43:41.307971Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by RoleBinding \\\"eks:certificate-controller/kube-system\\\" of Role \\\"eks:certificate-controller\\\" to User \\\"eks:certificate-controller\\\"\"}}",
      "timestamp": 1695329022421
    },
    {
      "id": "*********451",
      "message": "{\"kind\":\"Event\",\"apiVersion\":\"audit.k8s.io/v1\",\"level\":\"Metadata\",\"auditID\":\"9acfbded-797a-4bc1-bca6-cf6e8eddbba3\",\"stage\":\"ResponseComplete\",\"requestURI\":\"/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler?timeout=5s\",\"verb\":\"get\",\"user\":{\"username\":\"system:kube-scheduler\",\"groups\":[\"system:authenticated\"]},\"sourceIPs\":[\"172.16.111.18\"],\"userAgent\":\"kube-scheduler/v1.24.16 (linux/amd64) kubernetes/af930c1/leader-election\",\"objectRef\":{\"resource\":\"leases\",\"namespace\":\"kube-system\",\"name\":\"kube-scheduler\",\"apiGroup\":\"coordination.k8s.io\",\"apiVersion\":\"v1\"},\"responseStatus\":{\"metadata\":{},\"code\":200},\"requestReceivedTimestamp\":\"2023-09-21T20:43:42.331217Z\",\"stageTimestamp\":\"2023-09-21T20:43:42.338897Z\",\"annotations\":{\"authorization.k8s.io/decision\":\"allow\",\"authorization.k8s.io/reason\":\"RBAC: allowed by ClusterRoleBinding \\\"system:kube-scheduler\\\" of ClusterRole \\\"system:kube-scheduler\\\" to User \\\"system:kube-scheduler\\\"\"}}",
      "timestamp": 1695329022421
    }, ...
  ],
  "logGroup": "/aws/eks/*****-prod-manager/cluster",
  "logStream": "kube-apiserver-audit-******",
  "messageType": "DATA_MESSAGE",
  "owner": "******",
  "source_type": "stdin",
  "subscriptionFilters": [
    "AllowAll"
  ],
  "timestamp": "2023-09-21T20:44:48.994Z"
}

Output

{
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "6134a20d-7699-4311-bfe2-8eb8185a5dec",
  "id": "*********448",
  "kind": "Event",
  "level": "Metadata",
  ...
  "userAgent": "eks-certificates-controller/v0.0.0 (linux/amd64) kubernetes/$Format",
  "verb": "update"
}
{
  "apiVersion": "audit.k8s.io/v1",
  "auditID": "9acfbded-797a-4bc1-bca6-cf6e8eddbba3",
  "id": "*********451",
  "kind": "Event",
  "level": "Metadata",
  ...
  "userAgent": "kube-scheduler/v1.24.16 (linux/amd64) kubernetes/af930c1/leader-election",
  "verb": "get"
} ...

Results: The parsed data will explode and replace the original "message" field content.