Skip to main content

Destination Workflow

The DestinationWorkflow in Data Nadhi handles the final step: delivering processed logs to their intended destinations.
It gets data from the TransformationWorkflow and makes sure everything integrates properly with external systems.

[Github Repository]: data-nadhi-temporal-worker

Overview​

This workflow gets triggered as a child workflow from the TransformationWorkflow.
It fetches the integration targets and connector configs you need, and sends the data to the destination.

Destination Worker

Workflow Steps​

1. Fetch Integration Target​

Activity: fetch_integration_target

What it does:

  • Gets the target config using target_id
  • If target isn't found, the workflow throws an error
  • Provides target details to the next step

2. Fetch Integration Connector​

Activity: fetch_integration_connector

What it does:

  • Gets the connector config using the connector ID from the target
  • Decrypts sensitive credentials securely
  • If connector or credentials aren't found, throw an error
  • Provides connector details to the destination activity

3. Send to Destination​

Activity: send_to_destination

What it does:

  • Uses target and decrypted connector data to send logs/events
  • Handles failures here and retries according to the default retry policy

4. Exit​

  • Workflow completes successfully once data gets sent to the destination
  • Failure at any step logs detailed info to MinIO for traceability

Failure Handling​

All activity failures get logged to MinIO. Each log includes:

  • Exception type and message
  • Stack trace
  • Input data that caused the failure
  • Target and connector IDs

This keeps things transparent and makes troubleshooting easier if there are delivery issues.

Summary​

The DestinationWorkflow guarantees reliable delivery of processed logs to external systems by:

  • Validating targets and connectors before sending
  • Securely handling credentials in the connector fetch activity
  • Handling errors gracefully and logging all failures to MinIO

This makes sure every log reaches its intended destination securely and reliably.