till/managarten
1
0
Fork 0
mirror of https://github.com/Memo-2023/mana-monorepo.git synced 2026-05-14 20:21:09 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

fix(infra): simplify PostgreSQL backup to pg_dumpall + pg_basebackup

Browse source 
pgBackRest as Docker sidecar was overly complex (needs shared WAL
directory, stanza management, special entrypoint). Replace with a
simpler, proven approach using native PostgreSQL tools:

Backup container (postgres:16-alpine):
- Hourly: pg_dumpall | gzip (all databases as SQL, ~2 day retention)
- Daily 03:00: pg_basebackup -Ft -z (physical backup, 30 day retention)
- Auto-cleanup of old backups
- Storage: /Volumes/ManaData/backups/postgres/

Also: Remove pgbackrest.conf, simplify postgresql.conf (remove WAL
archiving config, keep performance tuning + replication for basebackup)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Till JS 2026-03-24 11:39:20 +01:00
parent d3ae3841d9
commit fcd7c82ce4
4 changed files with 114 additions and 169 deletions
Download patch file Download diff file Expand all files Collapse all files

45
docker-compose.macmini.yml
View file

@ -39,27 +39,46 @@ services:
postgres postgres
-c 'config_file=/etc/postgresql/conf.d/custom.conf' -c 'config_file=/etc/postgresql/conf.d/custom.conf'
# PostgreSQL Backup with Point-in-Time Recovery # PostgreSQL Backup — hourly pg_dumpall + daily pg_basebackup
# Full backup daily at 03:00, differential every 6h, incremental every hour # Retention: 48 hourly dumps + 30 daily base backups
# Retention: 4 full backups + 14 differential
# Restore: see docs/POSTGRES_BACKUP.md # Restore: see docs/POSTGRES_BACKUP.md
postgres-backup: postgres-backup:
image: woblerr/pgbackrest:latest image: postgres:16-alpine
container_name: mana-infra-pgbackrest container_name: mana-infra-postgres-backup
restart: unless-stopped restart: unless-stopped
depends_on: depends_on:
postgres: postgres:
condition: service_healthy condition: service_healthy
environment: environment:
PGBACKREST_STANZA: mana PGHOST: postgres
PGBACKREST_REPO1_PATH: /var/lib/pgbackrest PGUSER: postgres
PGBACKREST_REPO1_RETENTION_FULL: "4" PGPASSWORD: ${POSTGRES_PASSWORD:-mana123}
PGBACKREST_PG1_HOST: postgres BACKUP_DIR: /backups
PGBACKREST_PG1_PORT: "5432" RETENTION_HOURLY: 48
PGBACKREST_PG1_USER: postgres RETENTION_DAILY: 30
PGBACKREST_PG1_PATH: /var/lib/postgresql/data
volumes: volumes:
- /Volumes/ManaData/backups/pgbackrest:/var/lib/pgbackrest - /Volumes/ManaData/backups/postgres:/backups
entrypoint: >
/bin/sh -c "
echo 'PostgreSQL Backup Service started';
echo 'Hourly: pg_dumpall (retention: 48)';
echo 'Daily 03:00: pg_basebackup (retention: 30)';
while true; do
HOUR=$$(date +%H);
TIMESTAMP=$$(date +%Y%m%d_%H%M%S);
echo \"[$$TIMESTAMP] Running hourly backup...\";
pg_dumpall -h postgres -U postgres | gzip > /backups/hourly_$$TIMESTAMP.sql.gz;
if [ \"$$HOUR\" = '03' ]; then
echo \"[$$TIMESTAMP] Running daily base backup...\";
mkdir -p /backups/base_$$TIMESTAMP;
pg_basebackup -h postgres -U postgres -D /backups/base_$$TIMESTAMP -Ft -z -P;
fi;
find /backups -name 'hourly_*.sql.gz' -mmin +$$((48*60)) -delete 2>/dev/null;
find /backups -name 'base_*' -maxdepth 1 -type d -mtime +30 -exec rm -rf {} + 2>/dev/null;
echo \"[$$TIMESTAMP] Backup complete. Sleeping 1h...\";
sleep 3600;
done
"
redis: redis:
image: redis:7-alpine image: redis:7-alpine

22
docker/postgres/pgbackrest.conf
View file

@ -1,22 +0,0 @@
[global]
# Repository location (inside pgbackrest container, mapped to host SSD)
repo1-path=/var/lib/pgbackrest
repo1-retention-full=4
repo1-retention-diff=14
# Compression
compress-type=zst
compress-level=3
# Parallelism
process-max=2
# Logging
log-level-console=info
log-level-file=detail
[mana]
pg1-path=/var/lib/postgresql/data
pg1-host=postgres
pg1-port=5432
pg1-user=postgres

24
docker/postgres/postgresql.conf
View file

@ -1,23 +1,19 @@
# PostgreSQL Configuration for WAL Archiving + pgBackRest # PostgreSQL Performance Tuning for Mac Mini (16GB RAM)
# Append to default postgresql.conf # Shared with ~70 other containers, so conservative settings
# WAL Archiving (required for Point-in-Time Recovery) # Performance
wal_level = replica
archive_mode = on
archive_command = 'pgbackrest --stanza=mana archive-push %p'
archive_timeout = 60
# WAL Settings (tuned for 16GB Mac Mini)
max_wal_senders = 3
max_wal_size = 1GB
min_wal_size = 80MB
# Performance (tuned for 16GB RAM, shared with other services)
shared_buffers = 512MB shared_buffers = 512MB
effective_cache_size = 2GB effective_cache_size = 2GB
work_mem = 16MB work_mem = 16MB
maintenance_work_mem = 128MB maintenance_work_mem = 128MB
# WAL Settings
max_wal_size = 1GB
min_wal_size = 80MB
# Replication (for pg_basebackup)
max_wal_senders = 3
# Logging # Logging
log_min_duration_statement = 1000 log_min_duration_statement = 1000
log_checkpoints = on log_checkpoints = on

192
docs/POSTGRES_BACKUP.md
View file

@ -1,75 +1,78 @@
# PostgreSQL Backup mit pgBackRest # PostgreSQL Backup
> Point-in-Time Recovery (PITR) für alle Mana-Datenbanken > Stündliche Dumps + tägliche Base-Backups für alle Mana-Datenbanken
## Übersicht ## Übersicht
| Aspekt | Vorher (pg_dumpall) | Jetzt (pgBackRest) | | Aspekt | Vorher | Jetzt |
|--------|--------------------|--------------------| |--------|--------|-------|
| **Recovery Point** | Letzter Dump (24h Verlust möglich) | Bis auf die letzte Sekunde | | **Frequenz** | 1x täglich (03:00) | Stündlich + täglich |
| **Backup-Typ** | Nur Full Dump | Full + Differential + Incremental | | **Max. Datenverlust** | 24 Stunden | 1 Stunde |
| **Kompression** | Keine | Zstandard (zst) | | **Backup-Typ** | Nur pg_dumpall | pg_dumpall (stündlich) + pg_basebackup (täglich) |
| **Parallelisierung** | Nein | Ja (2 Threads) | | **Kompression** | Keine | gzip |
| **Validierung** | Keine | Automatisch | | **Retention** | 30 Tage | 48h Dumps + 30 Tage Base-Backups |
| **Automatisch** | LaunchAgent Cron | Docker Container (always running) |
## Architektur ## Architektur
``` ```
PostgreSQL (mana-infra-postgres) PostgreSQL (mana-infra-postgres)
│ │
├── WAL-Archivierung → pgBackRest empfängt WAL-Segmente └── postgres-backup Container (mana-infra-postgres-backup)
│ ├── Jede Stunde → pg_dumpall | gzip (alle DBs als SQL)
└── Scheduled Backups: └── Täglich 03:00 → pg_basebackup -Ft -z (physisches Backup)
├── 03:00 → Full Backup (alle Daten)
├── 09:00, 15:00, 21:00 → Differential (nur Änderungen seit Full)
└── Jede Stunde → Incremental (nur Änderungen seit letztem Backup)
``` ```
## Backup-Speicherort ## Backup-Speicherort
``` ```
/Volumes/ManaData/backups/pgbackrest/ /Volumes/ManaData/backups/postgres/
├── archive/ # WAL-Archive (kontinuierlich) ├── hourly_20260324_140000.sql.gz # SQL Dump (stündlich)
└── backup/ # Full + Diff + Incr Backups ├── hourly_20260324_150000.sql.gz
├── ...
├── base_20260324_030000/ # Base Backup (täglich)
│ ├── base.tar.gz
│ └── pg_wal.tar.gz
└── base_20260323_030000/
``` ```
**Retention:** 4 Full Backups + 14 Differentials (~4 Wochen Rückblick) **Retention:**
- Stündliche Dumps: 48 Stück (~2 Tage)
- Tägliche Base-Backups: 30 Stück (~1 Monat)
## Befehle ## Container-Status prüfen
### Status prüfen
```bash ```bash
# Backup-Info anzeigen # Container Status
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info docker ps --filter name=mana-infra-postgres-backup
# Letzte Backups auflisten # Logs (letzte Backups)
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info --output=json | jq '.[] | .backup[] | {label, type, timestamp_start, database_size}' docker logs mana-infra-postgres-backup --tail 20
# Backup-Größe
du -sh /Volumes/ManaData/backups/postgres/
ls -lah /Volumes/ManaData/backups/postgres/hourly_*.sql.gz | tail -5
``` ```
### Manuelles Backup ## Wiederherstellung
### Szenario 1: Aus stündlichem Dump (SQL)
```bash ```bash
# Full Backup (manuell) # 1. Gewünschten Dump finden
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=full backup ls -la /Volumes/ManaData/backups/postgres/hourly_*.sql.gz
# Differential Backup # 2. Alle Datenbanken wiederherstellen
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=diff backup gunzip -c /Volumes/ManaData/backups/postgres/hourly_20260324_140000.sql.gz \
| docker exec -i mana-infra-postgres psql -U postgres
# Incremental Backup # 3. Oder einzelne Datenbank:
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=incr backup gunzip -c /Volumes/ManaData/backups/postgres/hourly_20260324_140000.sql.gz \
| grep -A 99999 'connect chat_db' | grep -B 99999 'connect ' \
| docker exec -i mana-infra-postgres psql -U postgres -d chat_db
``` ```
### Backup validieren ### Szenario 2: Aus Base-Backup (physisch)
```bash
# Verify prüft Integrität aller Backups
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana verify
```
## Wiederherstellung (Restore)
### Szenario 1: Komplette Wiederherstellung (letzter Stand)
```bash ```bash
# 1. PostgreSQL stoppen # 1. PostgreSQL stoppen
@ -78,103 +81,52 @@ docker compose -f docker-compose.macmini.yml stop postgres
# 2. Altes Datenverzeichnis sichern # 2. Altes Datenverzeichnis sichern
sudo mv /Volumes/ManaData/postgres /Volumes/ManaData/postgres.broken sudo mv /Volumes/ManaData/postgres /Volumes/ManaData/postgres.broken
# 3. Neues Verzeichnis erstellen # 3. Neues Verzeichnis aus Backup erstellen
sudo mkdir -p /Volumes/ManaData/postgres sudo mkdir -p /Volumes/ManaData/postgres
sudo chown 70:70 /Volumes/ManaData/postgres cd /Volumes/ManaData/postgres
sudo tar xzf /Volumes/ManaData/backups/postgres/base_20260324_030000/base.tar.gz
sudo tar xzf /Volumes/ManaData/backups/postgres/base_20260324_030000/pg_wal.tar.gz -C pg_wal/
sudo chown -R 70:70 /Volumes/ManaData/postgres
# 4. Restore ausführen # 4. PostgreSQL starten
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana restore
# 5. PostgreSQL starten
docker compose -f docker-compose.macmini.yml start postgres docker compose -f docker-compose.macmini.yml start postgres
# 6. Prüfen # 5. Prüfen
docker exec mana-infra-postgres psql -U postgres -c "\l" docker exec mana-infra-postgres psql -U postgres -c "\l"
``` ```
### Szenario 2: Point-in-Time Recovery (z.B. auf 14:59 Uhr) ### Szenario 3: Einzelne Datenbank aus Dump
```bash ```bash
# 1. PostgreSQL stoppen # Chat-DB wiederherstellen (Drop + Recreate)
docker compose -f docker-compose.macmini.yml stop postgres docker exec mana-infra-postgres psql -U postgres -c "DROP DATABASE IF EXISTS chat_db;"
docker exec mana-infra-postgres psql -U postgres -c "CREATE DATABASE chat_db;"
# 2. Altes Datenverzeichnis sichern gunzip -c /Volumes/ManaData/backups/postgres/hourly_20260324_140000.sql.gz \
sudo mv /Volumes/ManaData/postgres /Volumes/ManaData/postgres.broken | docker exec -i mana-infra-postgres psql -U postgres -d chat_db
# 3. Neues Verzeichnis erstellen
sudo mkdir -p /Volumes/ManaData/postgres
sudo chown 70:70 /Volumes/ManaData/postgres
# 4. PITR Restore auf bestimmten Zeitpunkt
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana \
--type=time \
--target="2026-03-24 14:59:00+01" \
restore
# 5. PostgreSQL starten
docker compose -f docker-compose.macmini.yml start postgres
``` ```
### Szenario 3: Einzelne Datenbank wiederherstellen ## Manuelles Backup
pgBackRest stellt immer den gesamten PostgreSQL-Cluster wieder her. Für einzelne Datenbanken:
```bash ```bash
# 1. Restore in temporäres Verzeichnis # Sofortiger Dump
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana \ docker exec mana-infra-postgres pg_dumpall -U postgres | gzip > /Volumes/ManaData/backups/postgres/manual_$(date +%Y%m%d_%H%M%S).sql.gz
--pg1-path=/tmp/pg-restore \
restore
# 2. Temporären PostgreSQL starten und pg_dump auf einzelne DB # Sofortiges Base-Backup
docker run --rm -v /tmp/pg-restore:/var/lib/postgresql/data \ mkdir -p /Volumes/ManaData/backups/postgres/manual_base_$(date +%Y%m%d)
postgres:16-alpine pg_dump -U postgres chat_db > /tmp/chat_db.sql docker exec mana-infra-postgres pg_basebackup -U postgres -D /tmp/backup -Ft -z -P
docker cp mana-infra-postgres:/tmp/backup /Volumes/ManaData/backups/postgres/manual_base_$(date +%Y%m%d)
# 3. In laufende DB importieren
cat /tmp/chat_db.sql | docker exec -i mana-infra-postgres psql -U postgres chat_db
``` ```
## Monitoring
### In Grafana (geplant)
pgBackRest exportiert keine nativen Prometheus-Metriken. Monitoring via:
```bash
# Health-Check in health-check.sh hinzufügen:
LAST_BACKUP=$(docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info --output=json 2>/dev/null | jq -r '.[0].backup[-1].timestamp_stop // "never"')
echo "Last backup: $LAST_BACKUP"
```
### Alerting
Der Health-Check (`scripts/mac-mini/health-check.sh`) kann prüfen ob das letzte Backup älter als 24h ist und einen Telegram-Alert senden.
## Erste Einrichtung ## Erste Einrichtung
Beim ersten Start auf dem Server:
```bash ```bash
# 1. Backup-Verzeichnis erstellen # Backup-Verzeichnis erstellen
sudo mkdir -p /Volumes/ManaData/backups/pgbackrest sudo mkdir -p /Volumes/ManaData/backups/postgres
sudo chown 999:999 /Volumes/ManaData/backups/pgbackrest
# 2. Container starten # Container starten
docker compose -f docker-compose.macmini.yml up -d postgres postgres-backup docker compose -f docker-compose.macmini.yml up -d postgres-backup
# 3. Stanza erstellen (einmalig) # Prüfen ob Backup läuft
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana stanza-create docker logs mana-infra-postgres-backup --tail 5
# 4. Erstes Full Backup
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=full backup
# 5. Prüfen
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info
``` ```
## Konfigurationsdateien
| Datei | Zweck |
|-------|-------|
| `docker/postgres/postgresql.conf` | WAL-Archivierung + Performance-Tuning |
| `docker/postgres/pgbackrest.conf` | pgBackRest Stanza + Retention |
| `docker-compose.macmini.yml` | Container-Definition (postgres-backup) |