mirror of
https://github.com/Memo-2023/mana-monorepo.git
synced 2026-05-14 18:41:08 +02:00
feat(infra): add pgBackRest for PostgreSQL Point-in-Time Recovery
Replace simple pg_dumpall with pgBackRest PITR backup system. This enables recovery to any second, not just the last daily dump. Configuration: - docker/postgres/postgresql.conf: WAL archiving + performance tuning (shared_buffers=512MB, effective_cache_size=2GB for 16GB Mac Mini) - docker/postgres/pgbackrest.conf: stanza config + retention policy Docker (docker-compose.macmini.yml): - postgres: mount custom config, enable WAL archiving - postgres-backup: new pgBackRest container - Storage: /Volumes/ManaData/backups/pgbackrest - Retention: 4 full + 14 differential (~4 weeks) - Compression: Zstandard (zst) Backup Schedule: - 03:00 daily: Full backup - Every 6h: Differential (changes since last full) - Every hour: Incremental (changes since last backup) - Continuous: WAL archiving (every 60s) Documentation (docs/POSTGRES_BACKUP.md): - Complete restore procedures (full, PITR, single DB) - First-time setup instructions - Monitoring and alerting integration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Till JS
parent
979564540a
commit
39526918a3
4 changed files with 251 additions and 0 deletions
|
|
@ -27,6 +27,7 @@ services:
|
|||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-mana123}
|
||||
volumes:
|
||||
- /Volumes/ManaData/postgres:/var/lib/postgresql/data
|
||||
- ./docker/postgres/postgresql.conf:/etc/postgresql/conf.d/custom.conf:ro
|
||||
ports:
|
||||
- "5432:5432"
|
||||
healthcheck:
|
||||
|
|
@ -34,6 +35,31 @@ services:
|
|||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
command: >
|
||||
postgres
|
||||
-c 'config_file=/etc/postgresql/conf.d/custom.conf'
|
||||
|
||||
# PostgreSQL Backup with Point-in-Time Recovery
|
||||
# Full backup daily at 03:00, differential every 6h, incremental every hour
|
||||
# Retention: 4 full backups + 14 differential
|
||||
# Restore: see docs/POSTGRES_BACKUP.md
|
||||
postgres-backup:
|
||||
image: pgbackrest/pgbackrest:latest
|
||||
container_name: mana-infra-pgbackrest
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
PGBACKREST_STANZA: mana
|
||||
PGBACKREST_REPO1_PATH: /var/lib/pgbackrest
|
||||
PGBACKREST_REPO1_RETENTION_FULL: "4"
|
||||
PGBACKREST_PG1_HOST: postgres
|
||||
PGBACKREST_PG1_PORT: "5432"
|
||||
PGBACKREST_PG1_USER: postgres
|
||||
PGBACKREST_PG1_PATH: /var/lib/postgresql/data
|
||||
volumes:
|
||||
- /Volumes/ManaData/backups/pgbackrest:/var/lib/pgbackrest
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
|
|
|
|||
22
docker/postgres/pgbackrest.conf
Normal file
22
docker/postgres/pgbackrest.conf
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
[global]
|
||||
# Repository location (inside pgbackrest container, mapped to host SSD)
|
||||
repo1-path=/var/lib/pgbackrest
|
||||
repo1-retention-full=4
|
||||
repo1-retention-diff=14
|
||||
|
||||
# Compression
|
||||
compress-type=zst
|
||||
compress-level=3
|
||||
|
||||
# Parallelism
|
||||
process-max=2
|
||||
|
||||
# Logging
|
||||
log-level-console=info
|
||||
log-level-file=detail
|
||||
|
||||
[mana]
|
||||
pg1-path=/var/lib/postgresql/data
|
||||
pg1-host=postgres
|
||||
pg1-port=5432
|
||||
pg1-user=postgres
|
||||
23
docker/postgres/postgresql.conf
Normal file
23
docker/postgres/postgresql.conf
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
# PostgreSQL Configuration for WAL Archiving + pgBackRest
|
||||
# Append to default postgresql.conf
|
||||
|
||||
# WAL Archiving (required for Point-in-Time Recovery)
|
||||
wal_level = replica
|
||||
archive_mode = on
|
||||
archive_command = 'pgbackrest --stanza=mana archive-push %p'
|
||||
archive_timeout = 60
|
||||
|
||||
# WAL Settings (tuned for 16GB Mac Mini)
|
||||
max_wal_senders = 3
|
||||
max_wal_size = 1GB
|
||||
min_wal_size = 80MB
|
||||
|
||||
# Performance (tuned for 16GB RAM, shared with other services)
|
||||
shared_buffers = 512MB
|
||||
effective_cache_size = 2GB
|
||||
work_mem = 16MB
|
||||
maintenance_work_mem = 128MB
|
||||
|
||||
# Logging
|
||||
log_min_duration_statement = 1000
|
||||
log_checkpoints = on
|
||||
180
docs/POSTGRES_BACKUP.md
Normal file
180
docs/POSTGRES_BACKUP.md
Normal file
|
|
@ -0,0 +1,180 @@
|
|||
# PostgreSQL Backup mit pgBackRest
|
||||
|
||||
> Point-in-Time Recovery (PITR) für alle Mana-Datenbanken
|
||||
|
||||
## Übersicht
|
||||
|
||||
| Aspekt | Vorher (pg_dumpall) | Jetzt (pgBackRest) |
|
||||
|--------|--------------------|--------------------|
|
||||
| **Recovery Point** | Letzter Dump (24h Verlust möglich) | Bis auf die letzte Sekunde |
|
||||
| **Backup-Typ** | Nur Full Dump | Full + Differential + Incremental |
|
||||
| **Kompression** | Keine | Zstandard (zst) |
|
||||
| **Parallelisierung** | Nein | Ja (2 Threads) |
|
||||
| **Validierung** | Keine | Automatisch |
|
||||
|
||||
## Architektur
|
||||
|
||||
```
|
||||
PostgreSQL (mana-infra-postgres)
|
||||
│
|
||||
├── WAL-Archivierung → pgBackRest empfängt WAL-Segmente
|
||||
│
|
||||
└── Scheduled Backups:
|
||||
├── 03:00 → Full Backup (alle Daten)
|
||||
├── 09:00, 15:00, 21:00 → Differential (nur Änderungen seit Full)
|
||||
└── Jede Stunde → Incremental (nur Änderungen seit letztem Backup)
|
||||
```
|
||||
|
||||
## Backup-Speicherort
|
||||
|
||||
```
|
||||
/Volumes/ManaData/backups/pgbackrest/
|
||||
├── archive/ # WAL-Archive (kontinuierlich)
|
||||
└── backup/ # Full + Diff + Incr Backups
|
||||
```
|
||||
|
||||
**Retention:** 4 Full Backups + 14 Differentials (~4 Wochen Rückblick)
|
||||
|
||||
## Befehle
|
||||
|
||||
### Status prüfen
|
||||
|
||||
```bash
|
||||
# Backup-Info anzeigen
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info
|
||||
|
||||
# Letzte Backups auflisten
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info --output=json | jq '.[] | .backup[] | {label, type, timestamp_start, database_size}'
|
||||
```
|
||||
|
||||
### Manuelles Backup
|
||||
|
||||
```bash
|
||||
# Full Backup (manuell)
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=full backup
|
||||
|
||||
# Differential Backup
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=diff backup
|
||||
|
||||
# Incremental Backup
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=incr backup
|
||||
```
|
||||
|
||||
### Backup validieren
|
||||
|
||||
```bash
|
||||
# Verify prüft Integrität aller Backups
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana verify
|
||||
```
|
||||
|
||||
## Wiederherstellung (Restore)
|
||||
|
||||
### Szenario 1: Komplette Wiederherstellung (letzter Stand)
|
||||
|
||||
```bash
|
||||
# 1. PostgreSQL stoppen
|
||||
docker compose -f docker-compose.macmini.yml stop postgres
|
||||
|
||||
# 2. Altes Datenverzeichnis sichern
|
||||
sudo mv /Volumes/ManaData/postgres /Volumes/ManaData/postgres.broken
|
||||
|
||||
# 3. Neues Verzeichnis erstellen
|
||||
sudo mkdir -p /Volumes/ManaData/postgres
|
||||
sudo chown 70:70 /Volumes/ManaData/postgres
|
||||
|
||||
# 4. Restore ausführen
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana restore
|
||||
|
||||
# 5. PostgreSQL starten
|
||||
docker compose -f docker-compose.macmini.yml start postgres
|
||||
|
||||
# 6. Prüfen
|
||||
docker exec mana-infra-postgres psql -U postgres -c "\l"
|
||||
```
|
||||
|
||||
### Szenario 2: Point-in-Time Recovery (z.B. auf 14:59 Uhr)
|
||||
|
||||
```bash
|
||||
# 1. PostgreSQL stoppen
|
||||
docker compose -f docker-compose.macmini.yml stop postgres
|
||||
|
||||
# 2. Altes Datenverzeichnis sichern
|
||||
sudo mv /Volumes/ManaData/postgres /Volumes/ManaData/postgres.broken
|
||||
|
||||
# 3. Neues Verzeichnis erstellen
|
||||
sudo mkdir -p /Volumes/ManaData/postgres
|
||||
sudo chown 70:70 /Volumes/ManaData/postgres
|
||||
|
||||
# 4. PITR Restore auf bestimmten Zeitpunkt
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana \
|
||||
--type=time \
|
||||
--target="2026-03-24 14:59:00+01" \
|
||||
restore
|
||||
|
||||
# 5. PostgreSQL starten
|
||||
docker compose -f docker-compose.macmini.yml start postgres
|
||||
```
|
||||
|
||||
### Szenario 3: Einzelne Datenbank wiederherstellen
|
||||
|
||||
pgBackRest stellt immer den gesamten PostgreSQL-Cluster wieder her. Für einzelne Datenbanken:
|
||||
|
||||
```bash
|
||||
# 1. Restore in temporäres Verzeichnis
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana \
|
||||
--pg1-path=/tmp/pg-restore \
|
||||
restore
|
||||
|
||||
# 2. Temporären PostgreSQL starten und pg_dump auf einzelne DB
|
||||
docker run --rm -v /tmp/pg-restore:/var/lib/postgresql/data \
|
||||
postgres:16-alpine pg_dump -U postgres chat_db > /tmp/chat_db.sql
|
||||
|
||||
# 3. In laufende DB importieren
|
||||
cat /tmp/chat_db.sql | docker exec -i mana-infra-postgres psql -U postgres chat_db
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### In Grafana (geplant)
|
||||
|
||||
pgBackRest exportiert keine nativen Prometheus-Metriken. Monitoring via:
|
||||
|
||||
```bash
|
||||
# Health-Check in health-check.sh hinzufügen:
|
||||
LAST_BACKUP=$(docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info --output=json 2>/dev/null | jq -r '.[0].backup[-1].timestamp_stop // "never"')
|
||||
echo "Last backup: $LAST_BACKUP"
|
||||
```
|
||||
|
||||
### Alerting
|
||||
|
||||
Der Health-Check (`scripts/mac-mini/health-check.sh`) kann prüfen ob das letzte Backup älter als 24h ist und einen Telegram-Alert senden.
|
||||
|
||||
## Erste Einrichtung
|
||||
|
||||
Beim ersten Start auf dem Server:
|
||||
|
||||
```bash
|
||||
# 1. Backup-Verzeichnis erstellen
|
||||
sudo mkdir -p /Volumes/ManaData/backups/pgbackrest
|
||||
sudo chown 999:999 /Volumes/ManaData/backups/pgbackrest
|
||||
|
||||
# 2. Container starten
|
||||
docker compose -f docker-compose.macmini.yml up -d postgres postgres-backup
|
||||
|
||||
# 3. Stanza erstellen (einmalig)
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana stanza-create
|
||||
|
||||
# 4. Erstes Full Backup
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana --type=full backup
|
||||
|
||||
# 5. Prüfen
|
||||
docker exec mana-infra-pgbackrest pgbackrest --stanza=mana info
|
||||
```
|
||||
|
||||
## Konfigurationsdateien
|
||||
|
||||
| Datei | Zweck |
|
||||
|-------|-------|
|
||||
| `docker/postgres/postgresql.conf` | WAL-Archivierung + Performance-Tuning |
|
||||
| `docker/postgres/pgbackrest.conf` | pgBackRest Stanza + Retention |
|
||||
| `docker-compose.macmini.yml` | Container-Definition (postgres-backup) |
|
||||
Loading…
Add table
Add a link
Reference in a new issue