# Runbook

## Daily checks

```bash
docker compose ps
docker compose logs --since 1h --tail 100
./verify.sh
```

## Common problems

**Airflow will not start.** It needs a database and a Fernet key. Run
`airflow db migrate` once, and set `AIRFLOW__CORE__FERNET_KEY` in `.env`.

**ClickHouse eats disk.** It is designed to. Set a TTL on tables, or retention
rules, before you load production data.

**Trino cannot reach a source.** Every catalogue is defined in a properties file
under `etc/catalog/`. The default config has no sources wired - add one.

**Kafka consumer lags.** Check the consumer group with
`kafka-consumer-groups.sh --describe`. Lag that only grows means a consumer is
down or too slow, not that Kafka is broken.

**Superset login fails.** The default admin is created on first init. If you
skipped `superset init`, there is no user to log in as.

## Backups

State lives in named volumes:

```bash
docker run --rm -v quarry_clickhouse_data:/data -v "$PWD":/backup \
  alpine tar czf /backup/clickhouse-$(date +%F).tgz -C /data .
```

Stop the service first.

## Sizing

| Setup | Minimum |
|---|---|
| DuckDB + Polars | 4 GB RAM |
| Adding Airflow + ClickHouse | 8 GB RAM |
| Adding Trino + Superset | 12 GB RAM |
| Full stack with Kafka | 16 GB+ RAM |

## Services in this tier

- `airflow`
- `dagster`
- `prefect`
- `kafka`
- `debezium`
- `clickhouse`
- `trino`
- `superset`