Alerts & Channels

How Alerts and Channels Operate

Before configuring alerts and channels, it's important to understand how Observo AI's architecture influences their behavior and accessibility.

Observo AI employs a hierarchical architecture with two primary levels:

Organization Level - The Manager

The Organization represents the top-level entity managed by the Manager component, which serves as the central control hub deployed as a Kubernetes cluster. The Manager provides customers with tools to efficiently manage and optimize their observability data across their entire organization.

Site Level - The Data Plane

A Site functions as the data plane for an organization's telemetry data, comprising a suite of services deployed within a Kubernetes cluster. Each site operates as an independent data processing environment that manages sources, pipelines and destinations that handles telemetry data collection, optimization, enrichment and normalization.

How This Affects Alerts and Channels

This architecture creates distinct behaviors for alerts and channels:

Channels - Organization-Wide Resources

Channels are configured and managed at the organization level, making them shared resources available across all sites within your organization. This design allows for:

• Consistent notification pathways across multiple sites

• Centralized management of communication endpoints

• Reusable notification configurations

Alerts - Site-Bounded Resources

Alerts are created and bound at the individual site level, remaining specific to the data processing environment where they were configured. This means:

• Alerts created in one site are not visible from other sites

• Each site maintains its own alert configurations

• Alert management is localized to the relevant data processing environment

Understanding this distinction is crucial when setting up your monitoring and notification strategy across multiple sites within your organization.

Organization (Manager->K8s Cluster)
├── Channels 📢 (Multiple channels of different types can be defined)
│   ├── Email Channel
│   ├── Slack Channel
│   ├── Microsoft Teams Channel
│   ├── OpsGenie Channel
│   └── Webhook Channel
│
├── Site A (K8s Cluster)
│   ├── Alerts 🚨 (site-bounded)
│   │   ├── CPU Alert
│   │   └── Memory Alert
│   └── Uses Channels (Can be created at Site level)
│
├── Site B (K8s Cluster)
│   ├── Alerts 🚨 (site-bounded)
│   │   ├── Disk Alert
│   │   └── Network Alert
│   └── Uses Channels (Can be created at Site level)
│
└── Site C (K8s Cluster)
    ├── Alerts 🚨 (site-bounded)
    │   └── Service Alert
    └── Uses Channels (Can be created at Site level)

Alerts & Channels Directory

Alerts helps users to receive notifications by tapping on certain metrics or parameters of the application. Alerts are configured to use channels which provide delivery of notifications to the end-users.

Currently, these alert types exist:

• No Active Dataplane Instances: Allows users to add alerts when there are no active dataplane instances running, which could indicate service disruption or deployment issues.

• Source Data Ingestion Spike: Allows users to add alerts when source data ingestion volume suddenly increases beyond normal thresholds, which could indicate data quality issues or system stress.

• Source No Data: Allows users to add alerts when source is not emitting or pushing any data.

• Destination Errors: Allows users to add alerts when some error is occurring either configuring the destination or accessing it.

• Site Service Pod Restart: Allows users to add alerts when site service pods are restarting frequently, which could indicate underlying infrastructure problems or resource constraints.

• Dataplane High Memory Usage: Allows users to add alerts when memory usage crosses a certain threshold.

• Destination No Data: Allows users to add alerts when no data is being pushed to the destination

• Source Errors: Allows users to add alerts when some error is occurring either configuring the source, accessing the source or processing its data.

• Destination Data Ingestion Drop: Allows users to add alerts when destination data ingestion drops when compared to historical data.

• Source Data Ingestion Drop: Allows users to add alerts when source data ingestion drops when compared to historical data.

• Dataplane High CPU Usage: Allows users to add alerts when cpu usage crosses a certain threshold.

• Site Down: Allows users to add alerts when the site goes down.

• Pipeline Optimization Drop: Allows users to add alerts when pipeline performance optimization metrics decline, indicating reduced efficiency in data processing or potential bottlenecks in the pipeline.

Users can use the following channels for receiving notifications:

• Email: Sends alert notifications directly to specified email addresses, providing detailed alert information in email format.

• Slack: Integrates with Slack workspaces to deliver real-time alert notifications to designated channels or direct messages.

• Webhook: Sends HTTP POST requests to custom endpoints, allowing integration with external systems and custom alert handling workflows.

• Microsoft Teams: Integrates with Microsoft Teams to send alert notifications to team channels or chat conversations.

• OpsGenie: Integrates with Atlassian OpsGenie incident management platform to create, escalate, and manage alerts through their on-call and incident response workflows.

Alerts and channels are available across all sites.

Note: Alerts and channels are available across all sites.

Although alerts and channels are configured for a particular site, they have different visibility scopes within the same organization. Channels can be accessed and used across different sites within the organization. However, alerts are site-bounded - an alert created in one site will not be visible in other sites, even within the same organization.