Korrel

Open-source ingestion as code.

Korrel is a Git-native, self-hostable data ingestion framework designed for teams that need the simplicity of low-code configuration and the flexibility of full-code customization.

Define pipelines in version-controlled configuration files, run them locally or in a server environment, and move data from APIs, databases, files, and internal systems into warehouses, lakes, and analytics platforms.

Core Principles

• Git-native
• Self-hostable
• High-performance
• Incremental by default
• Low-code when possible
• Full-code when necessary
• Open-source forever

Example

korrel run customers

name: customers

source:
  type: rest
  base_url: https://api.example.com

destination:
  type: parquet
  path: ./data/customers

Korrel is built around one simple idea:

Data movement should be defined as code, versioned in Git, and easy to customize without rebuilding your entire platform.

Development

Install the repository hooks with:

uv run pre-commit install

Run the full hook set at any time with:

uv run pre-commit run --all-files