AWS Security Lake Integration

This guide provides a comprehensive overview for integrating AWS Security Lake with Observo. It covers the setup of custom sources in AWS Security Lake, configuring Observo destinations, and creating pipelines with OCSF serializers. Links to relevant resources are included where applicable.

Create a Custom Source in AWS Security Lake

Step 1: Log in to AWS Management Console

  1. Open the AWS Management Console in your browser.

  2. Navigate to the Security Lake service by searching for "Security Lake" in the search bar.

  3. Ensure you have already configured AWS Security Lake. If not, refer to AWS Security Lake Setup Guide.

Step 2: Configure a Custom Source

Custom Sources

  1. Choose “Add Custom Sources” to integrate third-party or external data.

  2. Provide the custom source name and a description.

Role ARN and External ID

  1. Security Lake will generate a Role ARN and External ID during the custom source configuration process.

  2. Note these details carefully as they are required for integration.

    Example ARN and External ID:

    • Role ARN: arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ROLE_NAME>

    • External ID: 123abc456def789ghi

  3. These details will be used later when setting up the destination in Observo.

Setup AWS Security Lake Destination in Observo

Step 1: Access the Observo Interface

  1. Log in to your Observo account.

  2. Navigate to the “Destinations” section from the left-hand panel.

Step 2: Add New Destination

  1. Click on “Add New Destination”.

  2. From the available destination types, select AWS Security Lake.

Step 3: Configure Destination Settings

For detailed information on each configuration parameter, refer to the AWS Security Lake Destination

  1. Role ARN and External ID:

    • Enter the Role ARN and External ID generated during the AWS Security Lake custom source setup.

    • Ensure the IAM role includes policies like s3:PutObject, s3:GetObject, and sts:AssumeRole.

  1. Region:

    • Specify the AWS region where your Security Lake is configured (e.g., us-east-1, eu-west-1). Refer to the AWS Regions and Endpoints for valid options.

  2. Destination Name:

    • Provide a unique name for this destination, such as SecurityLake-Logs.

  3. Parquet Format:

    • Define the Parquet format for the OCSF log events.

Step 4: Save and Verify Configuration

  1. Click on “Save” to finalize the setup.

  2. The destination should now appear in the Destinations list. Ensure it is marked as active and verify that it is correctly configured for log ingestion.

Detailed Steps to Create a Pipeline with OCSF Serializer in Observo

Step 1: Access the Pipelines Section

  1. Log into your Observo account.

  2. Navigate to the “Pipelines” section from the left-hand panel.

Step 2: Create a New Pipeline

  1. Click on “Create Pipeline”.

  2. From the dropdown, select the source of your logs (e.g., S3 bucket).

  3. In the Destination section, select the AWS Security Lake destination previously configured.

Step 3: Add OCSF Serializer and Other Transforms

  1. Click on the + sign to add transforms.

  2. Select and add transforms to optimize and enrich your data. Options may include:

    • Filtering: Apply regex or rules to reduce noise.

    • Deduplication: Remove redundant log entries.

  3. Add the OCSF Serializer to format your logs into OCSF standard.

    • Observo supports various serializers for different log formats.

    • If your required serializer is not listed, you can define a custom serializer.

Step 4: Deploy the Pipeline

  1. After configuring the pipeline (source, transforms, OCSF serializer, and destination), click “Deploy Pipeline”.

  2. Monitor logs and statuses to ensure data flows without issues.

Step 5: Verification

  1. Verify log entries in AWS Security Lake to ensure they are formatted as expected.

  2. Adjust mappings or configurations if discrepancies are identified.

Basic Troubleshooting Steps

Common Issues

  1. IAM Permissions:

    • Check that the IAM user or role has sufficient permissions for the integration.

  2. S3 Bucket Configuration:

    • Ensure the bucket is properly configured for log access and notifications.

  3. Log Format:

    • Confirm logs are in the correct format expected by both AWS Security Lake and Observo.

  4. Error Monitoring:

    • Use AWS CloudTrail to trace issues and monitor error messages.

Support Process

  1. Contact Observo Support for assistance with integration issues.

  2. Provide relevant logs, configurations, and error messages to expedite troubleshooting.

  3. Review FAQs and community forums for additional guidance.

Logs Written to OCSF and Top Use Cases for Integration

Log Types

  • PANW logs

  • VPC flow logs for network traffic monitoring

Top Use Cases

  1. Security Incident Response:

    • Monitor API calls and detect unauthorized access.

  2. Operational Monitoring:

    • Analyze traffic patterns and performance metrics.

  3. Compliance Audits:

    • Centralized log management for reporting and compliance.

Mechanism to Share Source Mapping to OCSF

  1. Use Observo's mapping features to convert various log formats into OCSF.

  2. Document and export mapping configurations for reuse.

  3. Share mappings through export options in Observo or internal documentation.

Maintenance of Components Deployed in the Customer's Account

  1. Regularly update OCSF versions and patches from Observo.

  2. Audit IAM roles and permissions periodically to ensure compliance.

  3. Optimize log ingestion pipelines for performance improvements.

How to Disable the Integration

  1. Navigate to the “Pipelines” section in Observo.

  2. Pause or delete pipelines using the AWS Security Lake destination.

  3. Optionally, remove or disable the destination in the “Destinations” section.

  4. Revoke IAM roles or permissions specific to the integration if no longer required.

By following this guide, you can successfully integrate AWS Security Lake with Observo, enabling advanced analytics and streamlined log management.

PreviousLookup TablesNextBest Practices

Last updated

Was this helpful?