Commit graph

6 commits

Author SHA1 Message Date
Till JS
62d9eb1f2b fix(infra): update status page, prometheus, and cloudflared for unified app
All web app subdomains (chat.mana.how, todo.mana.how, etc.) were removed
when the unified app launched, but monitoring configs still referenced them.
Update blackbox targets to use mana.how/route URLs, remove stale API backend
routes from cloudflared, clean up CORS origins, and fix status page generator
to handle route-based URLs.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 14:59:15 +02:00
Till JS
7ff72d6c2c feat(monitoring): auto-prune Docker + node_modules, 15-min disk check interval
- check-disk-space.sh: always prune dangling images, unused volumes, and
  build cache >7 days on every run (not just at critical threshold)
- check-disk-space.sh: auto-remove node_modules if found on server
  (never needed — Docker builds inside containers)
- disk-check launchd: reduce interval from 60min to 15min to catch
  disk issues faster (yesterday we hit 100% before hourly check caught it)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 20:14:13 +02:00
Till-JS
acc8de36ee feat(monitoring): add alerting stack and maintenance scripts
Medium priority stability improvements:

Alerting:
- Add vmalert for evaluating Prometheus alert rules
- Add alertmanager for alert routing and grouping
- Add alert-notifier service for Telegram/ntfy notifications
- Enable cadvisor scraping in prometheus config

Disk Monitoring:
- Add check-disk-space.sh for hourly disk monitoring
- Alert on 80% (warning) and 90% (critical) thresholds
- Auto-cleanup Docker when disk is critical
- Add com.manacore.disk-check.plist for LaunchD

Weekly Reports:
- Add weekly-report.sh for system health summary
- Includes: backup status, disk usage, container health,
  database stats, error log summary
- Runs every Sunday at 10 AM via LaunchD

Health Check Updates:
- Add checks for vmalert, alertmanager, alert-notifier

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-12 13:46:57 +01:00
Till-JS
3de2f25552 feat(mac-mini): add stability improvements
High priority stability features:
- Add all LaunchD plists to Git for version control
- Handle crash-looping containers (Restarting status) in ensure-containers.sh
- Add database backup script with daily/weekly rotation
- Add Docker log rotation setup (50MB max, 3 files per container)

New files:
- scripts/mac-mini/backup-databases.sh - Daily pg_dump with rotation
- scripts/mac-mini/setup-docker-logging.sh - Configure daemon.json
- scripts/mac-mini/launchd/*.plist - All 8 LaunchD service configs
- scripts/mac-mini/launchd/README.md - Documentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-12 13:33:44 +01:00
Till-JS
759b227355 🔧 fix(mac-mini): correct user path in LaunchD plist
Server user is 'mana', not 'till'.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-12 13:15:06 +01:00
Till-JS
2fe7f842c6 🔧 fix(mac-mini): add container recovery and update health check ports
- Add ensure-containers-running.sh to detect and auto-start stuck containers
- Add LaunchD plist for automatic container health checks every 5 minutes
- Update health-check.sh with correct ports (3031/5011 for todo, etc.)
- Update deploy.sh health checks to match docker-compose.macmini.yml
- Fix container name references (mana-infra-postgres instead of manacore-postgres)

This prevents 502 errors when containers get stuck in "Created" status.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-12 12:51:49 +01:00