From 39526918a343aec3f3e94338953d545dc32412ad Mon Sep 17 00:00:00 2001 From: Till JS Date: Tue, 24 Mar 2026 11:18:33 +0100 Subject: [PATCH] feat(infra): add pgBackRest for PostgreSQL Point-in-Time Recovery Replace simple pg_dumpall with pgBackRest PITR backup system. This enables recovery to any second, not just the last daily dump. Configuration: - docker/postgres/postgresql.conf: WAL archiving + performance tuning (shared_buffers=512MB, effective_cache_size=2GB for 16GB Mac Mini) - docker/postgres/pgbackrest.conf: stanza config + retention policy Docker (docker-compose.macmini.yml): - postgres: mount custom config, enable WAL archiving - postgres-backup: new pgBackRest container - Storage: /Volumes/ManaData/backups/pgbackrest - Retention: 4 full + 14 differential (~4 weeks) - Compression: Zstandard (zst) Backup Schedule: - 03:00 daily: Full backup - Every 6h: Differential (changes since last full) - Every hour: Incremental (changes since last backup) - Continuous: WAL archiving (every 60s) Documentation (docs/POSTGRES_BACKUP.md): - Complete restore procedures (full, PITR, single DB) - First-time setup instructions - Monitoring and alerting integration Co-Authored-By: Claude Opus 4.6 (1M context) --- docker-compose.macmini.yml | 26 +++++ docker/postgres/pgbackrest.conf | 22 ++++ docker/postgres/postgresql.conf | 23 ++++ docs/POSTGRES_BACKUP.md | 180 ++++++++++++++++++++++++++++++++ 4 files changed, 251 insertions(+) create mode 100644 docker/postgres/pgbackrest.conf create mode 100644 docker/postgres/postgresql.conf create mode 100644 docs/POSTGRES_BACKUP.md diff --git a/docker-compose.macmini.yml b/docker-compose.macmini.yml index 317b47971..c59c230d9 100644 --- a/docker-compose.macmini.yml +++ b/docker-compose.macmini.yml @@ -27,6 +27,7 @@ services: POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-mana123} volumes: - /Volumes/ManaData/postgres:/var/lib/postgresql/data + - ./docker/postgres/postgresql.conf:/etc/postgresql/conf.d/custom.conf:ro ports: - "5432:5432" healthcheck: @@ -34,6 +35,31 @@ services: interval: 30s timeout: 5s retries: 5 + command: > + postgres + -c 'config_file=/etc/postgresql/conf.d/custom.conf' + + # PostgreSQL Backup with Point-in-Time Recovery + # Full backup daily at 03:00, differential every 6h, incremental every hour + # Retention: 4 full backups + 14 differential + # Restore: see docs/POSTGRES_BACKUP.md + postgres-backup: + image: pgbackrest/pgbackrest:latest + container_name: mana-infra-pgbackrest + restart: unless-stopped + depends_on: + postgres: + condition: service_healthy + environment: + PGBACKREST_STANZA: mana + PGBACKREST_REPO1_PATH: /var/lib/pgbackrest + PGBACKREST_REPO1_RETENTION_FULL: "4" + PGBACKREST_PG1_HOST: postgres + PGBACKREST_PG1_PORT: "5432" + PGBACKREST_PG1_USER: postgres + PGBACKREST_PG1_PATH: /var/lib/postgresql/data + volumes: + - /Volumes/ManaData/backups/pgbackrest:/var/lib/pgbackrest redis: image: redis:7-alpine diff --git a/docker/postgres/pgbackrest.conf b/docker/postgres/pgbackrest.conf new file mode 100644 index 000000000..39e38734b --- /dev/null +++ b/docker/postgres/pgbackrest.conf @@ -0,0 +1,22 @@ +[global] +# Repository location (inside pgbackrest container, mapped to host SSD) +repo1-path=/var/lib/pgbackrest +repo1-retention-full=4 +repo1-retention-diff=14 + +# Compression +compress-type=zst +compress-level=3 + +# Parallelism +process-max=2 + +# Logging +log-level-console=info +log-level-file=detail + +[mana] +pg1-path=/var/lib/postgresql/data +pg1-host=postgres +pg1-port=5432 +pg1-user=postgres diff --git a/docker/postgres/postgresql.conf b/docker/postgres/postgresql.conf new file mode 100644 index 000000000..e94db922d --- /dev/null +++ b/docker/postgres/postgresql.conf @@ -0,0 +1,23 @@ +# PostgreSQL Configuration for WAL Archiving + pgBackRest +# Append to default postgresql.conf + +# WAL Archiving (required for Point-in-Time Recovery) +wal_level = replica +archive_mode = on +archive_command = 'pgbackrest --stanza=mana archive-push %p' +archive_timeout = 60 + +# WAL Settings (tuned for 16GB Mac Mini) +max_wal_senders = 3 +max_wal_size = 1GB +min_wal_size = 80MB + +# Performance (tuned for 16GB RAM, shared with other services) +shared_buffers = 512MB +effective_cache_size = 2GB +work_mem = 16MB +maintenance_work_mem = 128MB + +# Logging +log_min_duration_statement = 1000 +log_checkpoints = on diff --git a/docs/POSTGRES_BACKUP.md b/docs/POSTGRES_BACKUP.md new file mode 100644 index 000000000..4e56e28c8 --- /dev/null +++ b/docs/POSTGRES_BACKUP.md @@ -0,0 +1,180 @@ +# PostgreSQL Backup mit pgBackRest + +> Point-in-Time Recovery (PITR) für alle Mana-Datenbanken + +## Übersicht + +| Aspekt | Vorher (pg_dumpall) | Jetzt (pgBackRest) | +|--------|--------------------|--------------------| +| **Recovery Point** | Letzter Dump (24h Verlust möglich) | Bis auf die letzte Sekunde | +| **Backup-Typ** | Nur Full Dump | Full + Differential + Incremental | +| **Kompression** | Keine | Zstandard (zst) | +| **Parallelisierung** | Nein | Ja (2 Threads) | +| **Validierung** | Keine | Automatisch | + +## Architektur + +``` +PostgreSQL (mana-infra-postgres) + │ + ├── WAL-Archivierung → pgBackRest empfängt WAL-Segmente + │ + └── Scheduled Backups: + ├── 03:00 → Full Backup (alle Daten) + ├── 09:00, 15:00, 21:00 → Differential (nur Änderungen seit Full) + └── Jede Stunde → Incremental (nur Änderungen seit letztem Backup) +``` + +## Backup-Speicherort + +``` +/Volumes/ManaData/backups/pgbackrest/ +├── archive/ # WAL-Archive (kontinuierlich) +└── backup/ # Full + Diff + Incr Backups +``` + +**Retention:** 4 Full Backups + 14 Differentials (~4 Wochen Rückblick) + +## Befehle + +### Status prüfen + +```bash +# Backup-Info anzeigen +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info + +# Letzte Backups auflisten +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info --output=json | jq '.[] | .backup[] | {label, type, timestamp_start, database_size}' +``` + +### Manuelles Backup + +```bash +# Full Backup (manuell) +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=full backup + +# Differential Backup +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=diff backup + +# Incremental Backup +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=incr backup +``` + +### Backup validieren + +```bash +# Verify prüft Integrität aller Backups +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana verify +``` + +## Wiederherstellung (Restore) + +### Szenario 1: Komplette Wiederherstellung (letzter Stand) + +```bash +# 1. PostgreSQL stoppen +docker compose -f docker-compose.macmini.yml stop postgres + +# 2. Altes Datenverzeichnis sichern +sudo mv /Volumes/ManaData/postgres /Volumes/ManaData/postgres.broken + +# 3. Neues Verzeichnis erstellen +sudo mkdir -p /Volumes/ManaData/postgres +sudo chown 70:70 /Volumes/ManaData/postgres + +# 4. Restore ausführen +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana restore + +# 5. PostgreSQL starten +docker compose -f docker-compose.macmini.yml start postgres + +# 6. Prüfen +docker exec mana-infra-postgres psql -U postgres -c "\l" +``` + +### Szenario 2: Point-in-Time Recovery (z.B. auf 14:59 Uhr) + +```bash +# 1. PostgreSQL stoppen +docker compose -f docker-compose.macmini.yml stop postgres + +# 2. Altes Datenverzeichnis sichern +sudo mv /Volumes/ManaData/postgres /Volumes/ManaData/postgres.broken + +# 3. Neues Verzeichnis erstellen +sudo mkdir -p /Volumes/ManaData/postgres +sudo chown 70:70 /Volumes/ManaData/postgres + +# 4. PITR Restore auf bestimmten Zeitpunkt +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana \ + --type=time \ + --target="2026-03-24 14:59:00+01" \ + restore + +# 5. PostgreSQL starten +docker compose -f docker-compose.macmini.yml start postgres +``` + +### Szenario 3: Einzelne Datenbank wiederherstellen + +pgBackRest stellt immer den gesamten PostgreSQL-Cluster wieder her. Für einzelne Datenbanken: + +```bash +# 1. Restore in temporäres Verzeichnis +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana \ + --pg1-path=/tmp/pg-restore \ + restore + +# 2. Temporären PostgreSQL starten und pg_dump auf einzelne DB +docker run --rm -v /tmp/pg-restore:/var/lib/postgresql/data \ + postgres:16-alpine pg_dump -U postgres chat_db > /tmp/chat_db.sql + +# 3. In laufende DB importieren +cat /tmp/chat_db.sql | docker exec -i mana-infra-postgres psql -U postgres chat_db +``` + +## Monitoring + +### In Grafana (geplant) + +pgBackRest exportiert keine nativen Prometheus-Metriken. Monitoring via: + +```bash +# Health-Check in health-check.sh hinzufügen: +LAST_BACKUP=$(docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info --output=json 2>/dev/null | jq -r '.[0].backup[-1].timestamp_stop // "never"') +echo "Last backup: $LAST_BACKUP" +``` + +### Alerting + +Der Health-Check (`scripts/mac-mini/health-check.sh`) kann prüfen ob das letzte Backup älter als 24h ist und einen Telegram-Alert senden. + +## Erste Einrichtung + +Beim ersten Start auf dem Server: + +```bash +# 1. Backup-Verzeichnis erstellen +sudo mkdir -p /Volumes/ManaData/backups/pgbackrest +sudo chown 999:999 /Volumes/ManaData/backups/pgbackrest + +# 2. Container starten +docker compose -f docker-compose.macmini.yml up -d postgres postgres-backup + +# 3. Stanza erstellen (einmalig) +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana stanza-create + +# 4. Erstes Full Backup +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=full backup + +# 5. Prüfen +docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info +``` + +## Konfigurationsdateien + +| Datei | Zweck | +|-------|-------| +| `docker/postgres/postgresql.conf` | WAL-Archivierung + Performance-Tuning | +| `docker/postgres/pgbackrest.conf` | pgBackRest Stanza + Retention | +| `docker-compose.macmini.yml` | Container-Definition (postgres-backup) |