Skip to main content

Architecture Overview

Data Nadhi is a distributed, microservices-based data pipeline platform built for flexibility, scalability, and keeping things simple.
This doc gives you the high-level view and links to detailed pages for each component.

High-Level Architecture​

Data Nadhi is made up of multiple services that can be deployed independently. They work together to ingest, process, and deliver data wherever it needs to go.

Data Nadhi Architecture

Core Components​

Data Nadhi SDK​

  • Built as language-specific SDKs - each language gets its own repo.
  • Captures logs from your app and sends them securely to the Data Nadhi Server.
  • You'll need:
    • API Key for authentication
    • Log Config to decide which logs go to stdout and which go to the server
  • If something fails, the SDK logs the error (if you have that enabled) after getting a failure response from the server.
  • Available SDKs:
  • Detailed SDK Architecture →

Data Nadhi Server​

  • A Node.js-based API server built to scale horizontally.
  • What it does:
    • Validates your API Key and figures out which org/project you're working with
    • Checks if the pipeline you want is actually active
    • Pushes log events to Temporal's task-q queue
  • Repository
  • Detailed Server Architecture →

Data Nadhi Main Worker​

  • Grabs the workflow config for each pipeline and sets it up in a way that can be traversed.
  • Figures out which node to start from and pushes tasks to Temporal's task-q-transform.
  • Repository
  • Detailed Overview →

Data Nadhi Transformation Worker​

  • Goes through the workflow config and applies transformations to your data.
  • Handles different types of nodes:
    • transform nodes for changing data
    • conditional-branch nodes for routing
    • end nodes for pushing the transformed data to Temporal's task-q-destination.
  • Repository
  • Detailed Overview →

Data Nadhi Destination Worker​

  • Gets the destination and connector configs you've set up.
  • Picks the right destination sender engine for the job.
  • Sends your processed data to wherever it needs to go (databases, APIs, etc.).
  • Repository
  • Detailed Overview →

Utility Components​

Data Nadhi Dev​

  • Has everything you need for the development environment using Dev Containers.
  • Includes:
    • docker-compose with shared storage and all the infrastructure
    • Migration scripts for MongoDB and Postgres
  • Repository

Data Nadhi Internal Server​

  • A lightweight server for intermediate operations that the UI usually handles.
  • Makes testing easier when you don't have the frontend running.
  • Repository

💡 All components are modular and containerized, so the whole system is portable, can scale, and is ready for production.