Component guide
airflow
Workflow orchestration for data pipelines.
Quarry
in Quarry-Complete-+-Onboarding-300
airflow
Licence: Apache-2.0 | Language: Python | Source: https://github.com/apache/airflow
Role
Workflow orchestration
Why it's here
The default scheduler for data pipelines. DAGs as Python, retries, backfills and a UI that shows what actually ran.
Running it
Started by docker compose up -d on port 8080.
./fetch-sources.sh ./sources
tar xzf ./sources/airflow.tar.gz -C ./sources
This component ships inside the Quarry
stack. See pricing →