[mana-server-health] cron failed @ 2026-06-10 16:18:23 CEST #130

New issue

Closed

opened 2026-06-10 16:18:23 +02:00 by till · 1 comment

till commented

2026-06-10 16:18:23 +02:00

Owner

Job: mana-server-health
Host: Till-MacMini
Time: 2026-06-10 16:18:23 CEST

── Layer 1: Public via Cloudflare ─────────────
✗ auth.mana.how/api/v1/auth/healthz → 000000 (expected 200|404) — Cloudflared/Origin Issue
✓ sync2.mana.how/health → 200
✓ api.pageta.com/health → 404
✓ pageta.com → 200
✓ git.mana.how → 200
✓ share.mana.how/api/v1/manifests → 401
── Layer 2: SSH-Mux Port-Forwards ─────────────
✓ localhost:3001/healthz → 404
✓ localhost:3042/health → 200
✓ localhost:3030/ → 200
✓ localhost:3072/api/v1/manifests → 401
✓ localhost:3099/healthz → 200
── Layer 3: Colima-VM ─────────────────────────
✓ colima VM running (docker erreichbar)
── Layer 4: Docker-Container ──────────────────
✓ 112 Container running
── Layer 5: Public-Hostname-Sweep ─────────────
! npm.mana.how → 401
! s3.mana.how → 403
✓ alle 146 Public-Hostnames erreichbar
── Layer 6: Disk ──────────────────────────────
✓ / 15% (96Gi frei)
✓ /Volumes/ManaData 27% (2.7Ti frei)
── Layer 7: Externe SSD ───────────────────────
✓ /Volumes/ManaData gemountet
✓ Symlink .ollama → /Volumes/ManaData/ollama
✓ Symlink stt-models → /Volumes/ManaData/stt-models
✓ Symlink flux2 → /Volumes/ManaData/flux2

── Recovery-Hints ─────────────────────────────
1. SSH-Mux stuck (Layer 2 rot, Layer 3/4 grün):
     ssh -F /dev/null -O exit -S /Users/mana/.colima/_lima/colima/ssh.sock _
     # wartet auf neuen Forward-Request, baut Mux frisch auf

2. Container down (Layer 4 rot):
     docker --context colima logs <name> --tail 30
     # häufig: Postgres-Wait-Loop nach VM-Restart

3. Colima-VM nicht hoch (Layer 3 rot):
     rm -f /Volumes/ManaData/colima-disk/in_use_by
     nohup colima start >/tmp/colima-boot.log 2>&1 &

Auto-erstellt von cron-notify.sh (siehe mana/docs/playbooks/CRON_FAILURE_NOTIFICATIONS.md).

**Job:** `mana-server-health` **Host:** Till-MacMini **Time:** 2026-06-10 16:18:23 CEST ``` ── Layer 1: Public via Cloudflare ───────────── ✗ auth.mana.how/api/v1/auth/healthz → 000000 (expected 200|404) — Cloudflared/Origin Issue ✓ sync2.mana.how/health → 200 ✓ api.pageta.com/health → 404 ✓ pageta.com → 200 ✓ git.mana.how → 200 ✓ share.mana.how/api/v1/manifests → 401 ── Layer 2: SSH-Mux Port-Forwards ───────────── ✓ localhost:3001/healthz → 404 ✓ localhost:3042/health → 200 ✓ localhost:3030/ → 200 ✓ localhost:3072/api/v1/manifests → 401 ✓ localhost:3099/healthz → 200 ── Layer 3: Colima-VM ───────────────────────── ✓ colima VM running (docker erreichbar) ── Layer 4: Docker-Container ────────────────── ✓ 112 Container running ── Layer 5: Public-Hostname-Sweep ───────────── ! npm.mana.how → 401 ! s3.mana.how → 403 ✓ alle 146 Public-Hostnames erreichbar ── Layer 6: Disk ────────────────────────────── ✓ / 15% (96Gi frei) ✓ /Volumes/ManaData 27% (2.7Ti frei) ── Layer 7: Externe SSD ─────────────────────── ✓ /Volumes/ManaData gemountet ✓ Symlink .ollama → /Volumes/ManaData/ollama ✓ Symlink stt-models → /Volumes/ManaData/stt-models ✓ Symlink flux2 → /Volumes/ManaData/flux2 ── Recovery-Hints ───────────────────────────── 1. SSH-Mux stuck (Layer 2 rot, Layer 3/4 grün): ssh -F /dev/null -O exit -S /Users/mana/.colima/_lima/colima/ssh.sock _ # wartet auf neuen Forward-Request, baut Mux frisch auf 2. Container down (Layer 4 rot): docker --context colima logs <name> --tail 30 # häufig: Postgres-Wait-Loop nach VM-Restart 3. Colima-VM nicht hoch (Layer 3 rot): rm -f /Volumes/ManaData/colima-disk/in_use_by nohup colima start >/tmp/colima-boot.log 2>&1 & ``` _Auto-erstellt von `cron-notify.sh` (siehe `mana/docs/playbooks/CRON_FAILURE_NOTIFICATIONS.md`)._