CloudTrail

Observo natively does not support CloudTrail as a source. However, you can process CloudTrail logs through our observability pipeline by following the steps, mentioned in the document.

Purpose

The purpose of the CloudTrail integration is to enable comprehensive monitoring and analysis of AWS account activity by collecting CloudTrail logs directly from S3 buckets. This setup supports streamlined data pipelines, security monitoring, compliance auditing, and analytics, allowing organizations to enhance observability and data-driven decision-making for their AWS environments.

Prerequisites

Before configuring CloudTrail log collection for Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:

  • AWS CloudTrail Setup:

    • CloudTrail must be enabled and actively logging events in your AWS account.

    • CloudTrail must be configured to deliver logs to an S3 bucket.

    • Note the S3 bucket name and prefix where CloudTrail logs are stored.

  • AWS S3 Bucket:

    • An existing S3 bucket where CloudTrail logs are delivered.

    • S3 bucket must be configured to send notifications for new object creation events.

    • Ensure appropriate bucket permissions for Observo AI to read objects.

  • AWS SQS Queue:

    • An SQS queue must be created to receive S3 bucket notifications.

    • The queue must be configured with appropriate permissions to receive messages from S3.

    • Note the SQS queue URL and ARN for Observo AI configuration.

  • IAM Permissions:

    • Observo AI requires IAM credentials or role with permissions for:

      • S3: GetObject, ListBucket

      • SQS: ReceiveMessage, DeleteMessage, GetQueueAttributes

  • Network and Connectivity:

    • Configure VPC endpoints or internet gateway as needed.

    • Verify firewall rules allow outbound connections to AWS services.

Prerequisite
Description
Notes

CloudTrail Logs

CloudTrail enabled with S3 delivery

Note S3 bucket name and prefix for CloudTrail logs

S3 Bucket

Bucket containing CloudTrail logs

Must support event notifications

SQS Queue

Queue for S3 bucket notifications

Note queue URL and ARN

IAM Permissions

S3 read and SQS permissions

Required for reading logs and processing notifications

Network Connectivity

Observo AI access to AWS services

Configure VPC endpoints or internet access

Integration

The Integration section outlines the configuration steps for CloudTrail log collection. Follow these steps to set up the integration:

Step 1: Verify CloudTrail Configuration

  1. Go to the AWS Management Console.

  2. Navigate to CloudTrail → Trails.

  3. Select your CloudTrail trail and verify it is logging events.

  4. Note the S3 bucket name where logs are delivered.

  5. Note the S3 key prefix (if any) used for CloudTrail logs.

Step 2: Create an SQS Queue

  1. Navigate to SQS in the AWS Management Console.

  2. Click Create queue.

  3. Choose Standard queue type.

  4. Provide a unique queue name (e.g., observo-cloudtrail-queue).

  5. Configure queue settings

  6. Click Create queue.

  7. Note the Queue URL and Queue ARN for later use.

Step 3: Configure SQS Queue Policy

  1. In the SQS queue details page, click on the Access policy tab.

  2. Click Edit to modify the access policy.

Step 4: Configure S3 Bucket Notifications

  1. Navigate to S3 in the AWS Management Console.

  2. Select the S3 bucket where CloudTrail logs are stored.

  3. Click on the Properties tab.

  4. Scroll down to Event notifications and click Create event notification.

  5. Configure the event notification:

    • Event name: Provide a descriptive name (e.g., cloudtrail-to-observo)

    • Prefix: (Optional) Specify the CloudTrail prefix if logs are organized by prefix

    • Suffix: .json.gz (CloudTrail logs are gzip-compressed JSON files)

    • Event types: Select All object create events

    • Destination: Choose SQS queue

    • SQS queue: Select the queue created in Step 2

  6. Click Save changes.

  7. Verify the event notification is active.

Step 5: Create IAM User or Role for Observo AI

  1. Navigate to IAM in the AWS Management Console.

  2. Choose to create either an IAM User (with access keys) or an IAM Role (for cross-account access).

For IAM User:

  1. Click Users → Add users.

  2. Provide a username (e.g., observo-cloudtrail-reader).

  3. Select Access key - Programmatic access.

  4. Click Next: Permissions.

For IAM Role:

  1. Click Roles → Create role.

  2. Select Another AWS account as the trusted entity.

  3. Enter Observo AI's AWS account ID (obtain from Observo AI support).

  4. Click Next: Permissions.

  5. Click Create policy and add the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ReadAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::<CloudTrail-Bucket-Name>",
        "arn:aws:s3:::<CloudTrail-Bucket-Name>/*"
      ]
    },
    {
      "Sid": "SQSAccess",
      "Effect": "Allow",
      "Action": [
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage",
        "sqs:GetQueueAttributes"
      ],
      "Resource": "arn:aws:sqs:<Region>:<Account-ID>:<Queue-Name>"
    }
  ]
}

  1. Replace the placeholders with your specific values.

  2. Name the policy (e.g., ObservoCloudTrailS3Access).

  3. Attach the policy to the user or role.

  4. For IAM User: Complete the user creation and save the access key ID and secret access key.

  5. For IAM Role: Complete the role creation and note the Role ARN.

Step 6: Configure Observo AI AWS S3 Source

  1. Log in to the Observo AI platform.

  2. Navigate to the Sources tab.

  3. Click Add Source and select Create New.

  4. Choose AWS S3 from the list of available sources.

  5. Configure the source with the details mentioned in the AWS S3 Source .

Step 7: Verify Data Ingestion

  1. In Observo AI, navigate to the Data or Logs section.

  2. Filter by the source name created in Step 6.

  3. Verify that CloudTrail events are appearing in near real-time.

  4. Check for proper parsing and field extraction of CloudTrail JSON structure.

  5. Monitor the SQS queue in AWS Console to ensure messages are being processed and deleted.

Error Handling

This section provides guidance on common errors and issues that may occur during CloudTrail log collection and their resolutions.

1. S3 Access Denied Errors

Cause: Observo AI IAM user or role lacks necessary permissions to read from the S3 bucket.

Resolution:

  • Verify the IAM policy attached to the user/role includes s3:GetObject and s3:ListBucket permissions.

  • Ensure the S3 bucket ARN and bucket/* ARN are correctly specified in the policy.

  • Check S3 bucket policies for any explicit deny statements that might block access.

  • Verify the bucket name is spelled correctly and exists in the specified region.

  • If using server-side encryption, ensure the IAM user/role has permissions to use the KMS key.

2. SQS Access Denied

Cause: Observo AI IAM credentials lack permissions to receive or delete messages from the SQS queue.

Resolution:

  • Verify the IAM policy includes sqs:ReceiveMessage, sqs:DeleteMessage, and sqs:GetQueueAttributes permissions.

  • Ensure the SQS queue ARN in the policy matches the queue being used.

  • Check the SQS queue access policy allows the IAM user/role to access it.

  • Verify the queue URL is correct and the queue exists in the specified region.

3. No Messages in SQS Queue

Cause: S3 bucket notifications are not configured correctly or CloudTrail is not writing logs.

Resolution:

  • Verify CloudTrail is enabled and actively logging events.

  • Check S3 bucket event notification configuration is active.

  • Verify the event notification prefix and suffix match CloudTrail log paths (.json.gz).

  • Test by uploading a test file to the S3 bucket and checking if SQS receives a message.

  • Review CloudTrail logs in S3 to confirm new files are being created.

  • Check S3 bucket notification permissions in the SQS queue policy.

Troubleshooting

Troubleshooting Steps

Verify CloudTrail Logging

  1. Navigate to CloudTrail → Trails in AWS Console.

  2. Select your trail and verify it shows as Logging: ON.

  3. Check the S3 bucket field to confirm logs are being delivered.

  4. Navigate to the S3 bucket and verify recent log files exist.

  5. Download a sample log file and verify it contains valid CloudTrail events.

Check S3 Bucket Notifications

  1. Navigate to S3 → Buckets → [Your Bucket] → Properties.

  2. Scroll to Event notifications section.

  3. Verify the notification is Active and correctly configured.

  4. Check the destination SQS queue ARN is correct.

  5. Verify the prefix and suffix filters match CloudTrail log patterns.

Monitor SQS Queue

  1. Navigate to SQS → Queues → [Your Queue] in AWS Console.

  2. Click Send and receive messages to manually check for messages.

  3. Review queue metrics.

Review Observo AI Logs

  1. Access Observo AI platform logs.

  2. Search for errors related to the CloudTrail S3 source.

  3. Look for connection errors, authentication failures, or parsing errors.

Best Practices

Implement monitoring to proactively identify issues:

  • CloudWatch Alarms:

    • SQS queue depth exceeding threshold

    • SQS age of oldest message

    • S3 bucket event notifications delivery failures

  • Observo AI Metrics:

    • Monitor ingestion rate and lag

    • Track source errors and failures

  • Data Quality Checks:

    • Verify CloudTrail events are being parsed correctly

    • Monitor for missing fields or malformed data

PreviousCloudflareNextCollector

Last updated

Was this helpful?