Setup Your First Pipeline

Introduction

This guide will walk you through the process of setting up your first pipeline in Observo, focusing on configuring new sources and destinations effectively.

Steps to Create a Pipeline

Step 1: Navigate to Pipelines

Begin by opening the Observo interface and navigating to the "Pipelines" section located on the left panel.

Step 2: Add a Pipeline

• In the Pipelines section, click on the "Add Pipeline --> Create New" button found at the top right corner of the page.

Step 3: Create and Add a Source

Since there are no pre-configured sources:

• Navigate to the “Add a Source” panel.

• Click on “Add New” and select a "Source Type" from the dropdown, preferably Observo Sample Logs Source to generate dummy logs.

• Fill in the required details (source type, credentials, configurations).

• Leave the Log parser and PE config settings as default.

• Save the source and select it for your pipeline.

Step 4: Create and Add a Destination

After configuring the source:

• Proceed to the “Add a Destination” panel.

• Click on “Create New” and select a Destination type from the dropdown, ideally the null destination for simple test purposes.

• Provide the necessary details for the destination (type, access settings, etc.),

• Save and select this destination for your pipeline.

Step 5: Add Transformations

If required to process the data:

• Visit the Transformations panel.

• Use the “+” icon to add necessary transformations such as filters, aggregation, and enrichment.

• Note: This step is optional and modifications can be made post-deployment as well.

Step 6: Configure and Deploy

Once all components are configured:

• Modify the pipeline name to reflect its purpose.

• Review all settings to ensure they are correct.

• Click the "Save as draft" button to save your pipeline configuration if you do not want to deploy right now.

• Deploy your pipeline by clicking the “Deploy” button, initiating data processing.

Step 7: Verify Deployment

To confirm the deployment:

• Navigate to the "Notifications" section to check for deployment success notifications.

• Monitor the pipeline status and data flow in the Observo interface.

Step 8: Pipeline Overview Page

To view and manage your pipeline:

• Click on the pipeline name in the Pipelines section to access the overview page.

• This page provides detailed information about the pipeline’s operation, optimization, and error management.

• Edit settings as necessary from this page by clicking on the "Edit" button.

Conclusion

Congratulations on setting up and deploying your first pipeline in Observo! This setup allows for efficient data handling and processing, enabling robust data analysis and management.