# Runbook

## Daily checks

```bash
docker compose ps
docker compose logs --since 1h --tail 100
./verify.sh
```

## Common problems

**Prometheus says target down.** Check the scrape config in
`prometheus.yml`. The target must be reachable from the Prometheus
container, which means the container name, not localhost.

**Prometheus fills the disk.** Retention defaults to 15 days. Set
`--storage.tsdb.retention.time` explicitly, or move long-term storage to
VictoriaMetrics or Thanos.

**No traces appearing in Jaeger.** The collector must be receiving on 4317
and forwarding to Jaeger. Check `otel-collector-config.yml` and the
collector logs.

**Alertmanager sends nothing.** Alerts must be routed. A receiver with no
matching route is silently dropped - check the route tree.

**Fluent Bit reads nothing.** The volume must mount the log directory you
want to tail, and the path in the config must match.

## Retention and sizing

| Signal | Growth |
|---|---|
| Metrics | ~1-2 bytes per sample per series |
| Logs | depends entirely on volume |
| Traces | proportional to request rate |

Estimate before you deploy, not after the disk fills.

## Backups

```bash
docker run --rm -v beacon_prometheus_data:/data -v "$PWD":/backup \
  alpine tar czf /backup/prometheus-$(date +%F).tgz -C /data .
```

## Services in this tier

- `prometheus`
- `alertmanager`
- `victoriametrics`
- `thanos`
- `fluentbit`
- `otel`
- `jaeger`
- `uptimekuma`
- `superset`