CrowdStrike S3

Integrate CrowdStrike as a data source in the pipeline. Configure the S3 bucket to send change notifications to an SQS queue. The Observo dataplane will then read messages from the SQS queue and fetch the corresponding objects from the S3 bucket.

Purpose

The Observo AI CrowdStrike S3 Source enables users to pull data from Amazon S3 buckets containing CrowdStrike telemetry data into Observo AI for analysis and processing. This source supports ingesting security data, such as endpoint activity logs, in formats like JSON, and is designed for integration with SIEM, threat-hunting, and other security tools. This integration helps organizations streamline their security data pipelines, enhance threat detection, and optimize data processing for observability and analytics.

Prerequisites

Before configuring the CrowdStrike S3 source in Observo AI, ensure the following requirements are met to facilitate seamless data ingestion:

  • Observo AI Platform Setup:

    • The Observo AI platform must be installed and operational, with support for CrowdStrike S3 as a data source.

    • Verify that the platform supports JSON data formats, as CrowdStrike typically stores data in JSON. Additional formats may require specific parser configurations.

  • CrowdStrike Account and S3/SQS Access:

    • An active CrowdStrike account with access to the S3 buckets and SQS queues containing CrowdStrike data is required. Contact your CrowdStrike team to obtain the S3 bucket URL and SQS queue URL.

    • The S3 bucket must be configured to send s3:ObjectCreated:* events to an SQS queue for notifications, either directly or via an SNS topic.

  • AWS Account and Permissions:

    • An active AWS account with access to the CrowdStrike-provided S3 buckets and SQS queues is required.

    • Required IAM permissions for the AWS credentials used by Observo AI:

      • For S3: s3:GetObject, s3:ListBucket.

      • For SQS: sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility, sqs:GetQueueAttributes, sqs:GetQueueUrl.

    • Contact your CrowdStrike team to obtain the necessary AWS access key and secret key for authentication.

  • Authentication:

    • Prepare one of the following authentication methods:

      • Auto Authentication: Use IAM roles, shared credentials, environment variables, or a JSON credentials file provided by CrowdStrike.

      • Manual Authentication: Provide the AWS access key and secret key obtained from CrowdStrike.

      • Secret Authentication: Use a stored secret within Observo AI’s secure storage for the AWS credentials.

  • Network and Connectivity:

    • Ensure Observo AI can communicate with AWS services (S3 and SQS endpoints, e.g., s3.us-east-1.amazonaws.com).

    • Check for proxy settings, firewall rules, or VPC endpoint configurations that may affect connectivity to AWS endpoints.

Prerequisite
Description
Notes

Observo AI Platform

Must be installed and support CrowdStrike S3 sources

Verify JSON support; additional parsers may be needed

CrowdStrike Account

Active account with S3/SQS access

Obtain S3 bucket and SQS queue URLs from CrowdStrike

AWS Account

Active account with S3/SQS access

Configure S3 events to SQS

IAM Permissions

Required for S3 and SQS access

See permissions list above

Authentication

Auto, Manual, or Secret

Prepare credentials from CrowdStrike

Network

Connectivity to AWS services

Check VPC endpoints, proxies, and firewalls

Integration

The Integration section outlines the configurations for the CrowdStrike S3 source. To configure CrowdStrike S3 as a source in Observo AI, follow these steps to set up and test the data flow:

  1. Log in to Observo AI:

    • Navigate to the Sources tab.

    • Click the Add Source button and select Create New.

    • Choose CrowdStrike S3 from the list of available sources to begin configuration.

  2. General Settings:

    • Name: A unique identifier for the source, such as crowdstrike-s3-source-1.

    • Description (Optional): Provide a description for the source.

    • Region: The AWS region of the CrowdStrike-provided S3 bucket

      Example

      us-east-1

    • Compression: Select the compression scheme used for decompressing objects retrieved from S3, such as Gzip. Default: None.

      Options (Select one)

      Automatically determine the compression schema

      Gzip

      Uncompressed

      ZSTD

  3. Authentication:

    • Access Key: Enter the AWS access key ID provided by CrowdStrike for accessing the S3 bucket and SQS queue.

    • Secret Access Key: Enter the AWS secret access key provided by CrowdStrike.

    • Assume Role (Optional): Enter the IAM role ARN to use for reading events from SQS and objects from S3, if applicable.

    • Credentials File Path (Optional): Specify the path to a credentials file if not using access key and secret for authentication.

    • Auth IMDS Connect Timeout Seconds (Optional): Connect timeout for Instance Metadata Service (IMDS). Default: Empty.

    • Auth IMDS Max Attempts (Optional): Number of IMDS retries for fetching tokens and metadata. Default: Empty.

    • Auth IMDS Read Timeout Seconds (Optional): Read timeout for IMDS. Default: Empty.

    • Auth Load Timeout Secs (Optional): Timeout for successfully loading credentials, in seconds. Default: Empty.

    • Auth Profile (Optional): The credentials profile to use from a provided credentials file. Default: Empty.

  4. SQS Configuration (Optional):

    • SQS Queue URL: The URL of the SQS queue to poll for bucket notifications, provided by CrowdStrike.

      Example

      https://sqs.us-east-1.amazonaws.com/123456789012/myQueueName

    • SQS Visibility Timeout Secs: Enter the visibility timeout to use for messages, in seconds. This controls how long a message is left unavailable after it is received. If a message is received, and takes longer than visibility_timeout_secs to process and delete the message from the queue, it is made available again for another consumer. This can happen if there is an issue between consuming a message and deleting it.

    • SQS Client Concurrency: Number of concurrent tasks to create for polling the queue for messages. Defaults to the number of available CPUs on the system. Should not typically need to be changed, but it can sometimes be beneficial to raise this value when there is a high rate of messages being pushed into the queue and the objects being fetched are small. In these cases, system resources may not be fully utilized without fetching more messages per second, as the SQS message consumption rate affects the S3 object retrieval rate.

      Example

      5

    • SQS Delete Message (False): Whether to delete the message once it is processed. It can be useful to set this to false for debugging or during the initial setup.

    • SQS Poll Secs: How long to wait while polling the queue for new messages, in seconds. Generally should not be changed unless instructed to do so, as if messages are available, they will always be consumed, regardless of the value of poll_secs.

    • SQS Poll Secs: How long to wait while polling the queue for new messages, in seconds. Generally should not be changed unless instructed to do so, as if messages are available, they will always be consumed, regardless of the value of poll_secs.

    • SQS TLS Options Ca File: Absolute path to an additional CA certificate file. The certificate must be in the DER or PEM (X.509) format. Additionally, the certificate can be provided as an inline string in PEM format.

      Example

      /path/to/certificate_authority.crt

    • SQS TLS Options Crt File: Absolute path to a certificate file used to identify this server. The certificate must be in DER, PEM (X.509), or PKCS#12 format. Additionally, the certificate can be provided as an inline string in PEM format. If this is set, and is not a PKCS#12 archive, key_file must also be set.

      Example

      /path/to/host_certificate.crt

    • SQS TLS Options Key File: Absolute path to a private key file used to identify this server. The key must be in DER or PEM (PKCS#8) format. Additionally, the key can be provided as an inline string in PEM format.

      Example

      /path/to/host_certificate.key

    • SQS TLS Options Key Pass: Passphrase used to unlock the encrypted key file. This has no effect unless key_file is set.

      Example

      ${KEY_PASS_ENV_VAR}

      PassWord1

  5. Multiline Settings (Optional):

    • Multiline Condition Pattern: Regular expression pattern that is used to determine whether or not more lines should be read. This setting must be configured in conjunction with mode.

      Examples

      ^[\s]+

      \\$

      ^(INFO | ERROR)

      ;$

    • Multiline Mode (Empty): Specifies how log lines are grouped. This setting must be configured with condition_pattern.

      Options
      Description

      Include +1

      Includes all lines matching the pattern and one additional line.

      Include Match

      Includes all lines matching the pattern. Useful for stack traces or continuation indicators.

      Stop Before

      Groups all lines up to and including the line that matches the pattern, which often marks the end of a message.

      Stop After

      Groups all lines up to and including the line that matches the pattern, which often marks the end of a message.

    • Multiline Start Pattern: Regular expression pattern that is used to match the start of a new message.

      Examples

      ^[\s]+

      \\$

      ^(INFO | ERROR)

      ;$

    • Multiline Timeout Ms: The maximum amount of time to wait for the next additional line, in milliseconds. Once this timeout is reached, the buffered message is guaranteed to be flushed, even if incomplete.

      Example

      1000

      600000

  6. TLS Options (Optional):

    • TLS Options CA File (Empty): Absolute path to an additional CA certificate file.

      Example

      /path/to/certificate_authority.crt

    • TLS Options Crt File (Empty): Absolute path to a certificate file used to identify this server.

      Example

      /path/to/host_certificate.crt

    • TLS Options Key File (Empty): Absolute path to a private key file used to identify this server.

      Example

      /path/to/host_certificate.key

    • TLS Options Key Passphrase (Empty): Passphrase to unlock the encrypted key file, if applicable.

      Example

      ${KEY_PASSWORD_ENV_VAR}

    • TLS Options Verify Hostname: Disabled by default. Enable for hostname verification.

  7. Framing (Optional):

    • Framing Delimiter (Empty): The character that delimits byte sequences.

    • Framing Max Length (None): The maximum length of the byte buffer.

    • Framing Method (Empty): The framing method.

      Options
      Description

      Byte Frames

      Byte frames are passed through as-is according to the underlying I/O boundaries (for example, split between messages or stream segments).

      Character Delimited

      Byte frames which are delimited by a chosen character.

      Length Delimited

      Byte frames which are prefixed by an unsigned big-endian 32-bit integer indicating the length.

      Newline Delimited

      Byte frames which are delimited by a newline character.

      Octet Counting

      Byte frames according to the octet counting format.

    • Framing Newline Delimited Max Length: **(**None).

    • Framing Octet Counting Max Length: (None).

  8. Advanced Settings (Optional):

    • Endpoint: Custom endpoint for use with AWS-compatible services.

Example

http://127.0.0.0:5000/path/to/service

  • Format: crowdstrike

  1. Parser Config:

    • Enable Source Log Parser: Disabled by default. Toggle to enable and select an appropriate parser for CrowdStrike data such as JSON parser from the Source Log Parser dropdown.

    • Add additional parsers as needed for specific data formats.

  2. Pattern Extractor:

    • Refer to Observo AI’s Pattern Extractor documentation for details on configuring pattern-based data extraction.

  3. Archival Destination:

    • Toggle Enable Archival on Source Switch to enable

    • Under Archival Destination, select from the list of Archival Destinations (Required)

  4. Save and Test Configuration:

    • Save the configuration settings in Observo AI.

    • Send sample data from the S3 bucket and verify that it is ingested into Observo AI, checking the Analytics tab for data flow.

Example Scenarios

Enerex, a fictitious utility company deploying millions of smart meters, has integrated CrowdStrike for endpoint security across their IT infrastructure. They want to ingest CrowdStrike telemetry data (e.g., endpoint activity logs in JSON format) stored in an Amazon S3 bucket into Observo AI for threat detection and observability. The S3 bucket is in the us-east-1 region, and notifications are sent via an SQS queue. Authentication uses AWS IAM role assumption with CrowdStrike-provided credentials, and TLS is enabled for secure communication. The configuration ensures efficient ingestion of security data to monitor potential threats to their smart meter network.

Standard HTTP Source Setup

Here is a standard CrowdStrike S3 Source configuration example. Only the required sections and their associated field updates are displayed in the table below:

General Settings

Field
Value
Notes

Name

crowdstrike-s3-enerex-1

Unique identifier for the CrowdStrike S3 source, indicating Enerex security data ingestion.

Description

Ingest CrowdStrike endpoint telemetry from S3 for Enerex threat detection

Optional, provides context for the source’s purpose.

Region

us-east-1

AWS region of the CrowdStrike-provided S3 bucket, matching the bucket’s location.

Compression

Gzip

Compression scheme for decompressing S3 objects, as CrowdStrike data is typically Gzip-compressed JSON.

Authentication

Field
Value
Notes

Access Key

AKIAXYZ1234567890ABC

AWS access key ID provided by CrowdStrike for S3 and SQS access.

Secret Access Key

wJalrXUtnFEMI/1234567890abcdefEXAMPLEKEY

AWS secret access key provided by CrowdStrike, securely stored.

Assume Role

arn:aws:iam::987654321012:role/CrowdStrikeObservoRole

IAM role ARN for role assumption, allowing Observo AI to access S3/SQS. Uses Enerex’s AWS account ID.

Assume Role Region

us-east-1

Region for STS requests, matching the S3 bucket region.

Auth Profile

enerex-crowdstrike

Credentials profile for the provided credentials, stored in Observo AI’s secure storage.

SQS Configuration

Field
Value
Notes

SQS Queue URL

https://sqs.us-east-1.amazonaws.com/987654321012/crowdstrike-enerex-queue

URL of the SQS queue for S3 event notifications, provided by CrowdStrike.

SQS Visibility Timeout Secs

600

Timeout (in seconds) for message processing to prevent re-delivery during ingestion.

SQS Client Concurrency

5

Number of concurrent tasks for polling SQS, suitable for high message rates.

SQS Delete Message

False

Disabled for initial setup to aid debugging; set to True in production.

SQS Poll Secs

(Empty)

Default polling duration, as messages are consumed immediately if available.

SQS TLS Options Ca File

-----BEGIN CERTIFICATE----- MIID... (PEM format)

Inline PEM string for the CA certificate, ensuring secure SQS communication.

SQS TLS Options Crt File

-----BEGIN CERTIFICATE----- MIIC... (PEM format)

Inline PEM string for the client certificate to identify Observo AI.

SQS TLS Options Key File

-----BEGIN PRIVATE KEY----- MIIE... (PEM format)

Inline PEM string for the private key, securely stored.

SQS TLS Options Key Pass

CrowdStrikeKey2025!

Passphrase to unlock the encrypted key file.

TLS Options

Field
Value
Notes

TLS Options CA File

-----BEGIN CERTIFICATE----- MIID... (PEM format)

Inline PEM string for the CA certificate, ensuring secure S3 communication.

TLS Options Crt File

-----BEGIN CERTIFICATE----- MIIC... (PEM format)

Inline PEM string for the client certificate to identify Observo AI.

TLS Options Key File

-----BEGIN PRIVATE KEY----- MIIE... (PEM format)

Inline PEM string for the private key, securely stored.

TLS Options Key Pass

CrowdStrikeKey2025!

Passphrase to unlock the encrypted key file.

TLS Options Verify Hostname

True

Enabled to ensure the S3/SQS hostname matches the certificate.

Test Configuration:

  • Click “Save” to store the configuration settings in Observo.

  • Send sample CrowdStrike telemetry data to the S3 bucket, triggering SQS notifications. Verify ingestion by monitoring the Analytics tab in the Observo AI pipeline for event counts and throughput.

Notes:

  • Authentication: The Access Key and Secret Access Key are fictional but follow AWS standards. Enerex must obtain these from CrowdStrike, along with the Assume Role ARN for secure access. The Auth Profile (enerex-crowdstrike) is stored securely in Observo AI.

  • TLS Configuration: PEM certificate/key strings are placeholders; actual values must be provided by CrowdStrike or Enerex’s AWS setup, ensuring secure S3/SQS communication.

  • SQS Configuration: The SQS Queue URL and concurrency settings (SQS Client Concurrency: 5) are optimized for high-volume telemetry. SQS Delete Message: False aids initial debugging but should be enabled in production to prevent duplicate processing.

  • Network: Ensure firewall rules allow HTTPS access to AWS S3/SQS endpoints (s3.us-east-1.amazonaws.com, sqs.us-east-1.amazonaws.com) on port 443.

  • Troubleshooting: If issues occur (e.g., “Missing credentials” or “Bucket does not exist”), verify the S3 bucket name, SQS queue URL, IAM permissions, and TLS settings. Use the Analytics tab and AWS CloudWatch for diagnostics, as per the Troubleshooting section.

  • Resources: Refer to CrowdStrike Documentation and AWS Documentation (e.g., Amazon S3 Event Notifications) for additional guidance.

Troubleshooting

If issues arise with the CrowdStrike S3 source in Observo AI, use the following steps to diagnose and resolve them:

  • Verify Configuration Settings:

    • Ensure all fields, such as SQS Queue URL, Region, and Filename Filter, are correctly entered and match the CrowdStrike-provided AWS setup.

    • Confirm that the S3 bucket is configured to send s3:ObjectCreated:* events to the specified SQS queue, as set up by CrowdStrike.

  • Check Authentication:

    • Verify the authentication method:

      • For Auto authentication, ensure IAM roles, shared credentials, or environment variables are correctly configured with CrowdStrike-provided credentials.

      • For Manual authentication, check that the access key and secret key provided by CrowdStrike are valid.

      • For Secret authentication, confirm the secret is accessible in Observo AI’s secure storage.

    • Ensure the credentials have the required permissions:

      • S3: s3:GetObject, s3:ListBucket.

      • SQS: sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility, sqs:GetQueueAttributes, sqs:GetQueueUrl.

  • Validate Network Connectivity:

    • Check for firewall rules, VPC endpoint configurations, or proxy settings that may block access to AWS S3 and SQS endpoints.

    • Test connectivity using the AWS CLI with similar proxy configurations to verify access to the CrowdStrike-provided S3 bucket and SQS queue.

  • Common Error Messages:

    • "Inaccessible host": May indicate TLS version mismatches (e.g., TLS 1.3 issues) or DNS problems. Ensure the host supports the required TLS version and check DNS settings.

    • "Missing credentials": Verify that the authentication method is correctly configured. For IAM roles, ensure the role is assumed correctly.

    • "Bucket does not exist": Check the S3 bucket name provided by CrowdStrike and ensure there are no certificate validation issues. Consider disabling "Reject unauthorized certificates" or adding CA certificates if needed.

  • Monitor Logs and Data:

    • Verify that data is being ingested by monitoring the SQS queue and S3 bucket activity.

    • Use the Analytics tab in the targeted Observo AI pipeline to monitor data volume and ensure expected throughput.

    • Check Observo AI logs for errors or warnings related to data ingestion from the CrowdStrike S3 source.

Issue
Possible Cause
Resolution

Data not ingested

Incorrect SQS queue URL or bucket configuration

Verify S3 event notifications to SQS

Authentication errors

Invalid credentials or role

Check authentication method and permissions

Connectivity issues

Firewall or proxy blocking access

Test network connectivity and VPC endpoints

"Inaccessible host"

TLS or DNS issues

Ensure TLS compatibility and check DNS

"Missing credentials"

Authentication misconfiguration

Verify IAM roles or manual credentials

"Bucket does not exist"

Incorrect bucket name or certificate issues

Check bucket name and certificate settings

Resources

For additional guidance and detailed information, refer to the following resources:

CrowdStrike Documentation:

AWS Documentation:

PreviousCrowdStrike AlertsNextCisco eStreamer

Last updated

Was this helpful?