till/managarten
1
0
Fork 0
mirror of https://github.com/Memo-2023/mana-monorepo.git synced 2026-05-14 21:21:10 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
managarten/infrastructure
History
Till JS 384be93274 feat(gpu-box): healthchecks for glitchtip-worker, gpu-promtail, status-gen 
Three containers were running with no healthcheck — Docker showed them
as 'none', so an actual crash inside the container would only surface
once the process itself exited (and got restarted by restart-policy).
Added container-internal probes that don't depend on tools the image
doesn't ship:

- glitchtip-worker: bash + /dev/tcp/glitchtip-redis/6379 — confirms the
  Celery broker is reachable. Bare-metal probe, no extra deps.
- gpu-promtail: bash + /dev/tcp/loki/3100 — confirms the loki sink the
  worker is shipping to is reachable. Replaces the wget-based check
  that errored 'executable file not found' on every tick.
- status-page-gen: stat + date — confirms /output/status.json was
  rewritten in the last 3 min (script writes it every 60s). Catches
  the case where the apk-install loop wedges or the generator
  silently dies.

CMD-SHELL is /bin/sh which is dash on Debian-based images and dash
doesn't support /dev/tcp — used CMD form with explicit bash for the
two TCP probes.

photon stays without a healthcheck — pre-existing user container, not
in this compose file. Adding it would require a recreate which loses
the warm OSM cache.

After rollout: 17/20 GPU-Box containers healthy + 3 'none' (status-nginx,
glitchtip-redis, gpu-node-exporter — all standard upstream images
without built-in /health endpoints; their service is checked indirectly
via downstream consumers' healthchecks).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-07 15:29:04 +02:00
..
.env.gpu-box.example infra(gpu-box): commit GPU-Box compose to repo + Phase 2e docs 2026-05-07 13:28:49 +02:00
docker-compose.gpu-box.yml feat(gpu-box): healthchecks for glitchtip-worker, gpu-promtail, status-gen 2026-05-07 15:29:04 +02:00
README.md docs(infra): document photon.mana.how + cross-LAN workaround pattern 2026-05-07 14:45:34 +02:00

README.md

Infrastructure Compose Files

Compose-Stacks die nicht direkt am Mac-Mini hängen.

docker-compose.gpu-box.yml

Mana-Stack auf der Windows-GPU-Box (Hostname mana-server-gpu, IP 192.168.178.11) in WSL2/Docker. Bekam mit Phase 2 (Mai 2026) die nicht-zeitkritischen Hilfsdienste vom Mini abgegeben — siehe docs/PLAN_OPTION_C.md.

Was läuft drin

Service Port (LAN-internal/external via Tunnel) Rolle
grafana :8000 → grafana.mana.how Dashboards (Phase 2a)
forgejo :3041 → git.mana.how Git-Mirror (Phase 2b)
umami :8010 → stats.mana.how Web-Analytics (Phase 2b)
victoriametrics :9090 (intern) Metrics-Store (Phase 2c)
loki :3100 (intern) Log-Store (Phase 2c)
pushgateway, blackbox-exporter, vmalert, alertmanager, alert-notifier (intern) Metrics + Alerting (Phase 2c)
gpu-node-exporter, gpu-cadvisor, gpu-promtail (intern) Self-Monitoring (Phase 2c)
glitchtip + worker + dedizierte postgres + redis :8020 → glitchtip.mana.how Error-Tracking mit eigenem DB-Stack (Phase 2d)
status-page-gen, status-nginx :8090 → status.mana.how Status-Seite (Phase 2e)

Plus der bestehende photon-Container (Geocoder), der vor Phase 2 schon auf der Box existierte und unangetastet blieb.

Live-Files auf der GPU-Box

/srv/mana/
├── docker-compose.gpu-box.yml    ← Diese Datei (live deployment)
├── .env                           ← Secrets, gitignored — Vorlage in .env.gpu-box.example
├── monitoring/                    ← Configs (rsync von Mini bei Phase 2c)
│   ├── prometheus/                ← prometheus.yml mit 192.168.178.131-Targets, alerts.yml
│   ├── grafana/                   ← provisioning + dashboards
│   ├── loki/, blackbox/, alertmanager/, alert-notifier/, promtail-gpu/
├── forgejo-data/                  ← Forgejo /data bind-mount (rsync von Mini bei Phase 2b)
└── source/                        ← Sparse mana-monorepo-clone
                                     für status-page-gen + zukünftige Scripts
                                     git pull stündlich via systemd timer mana-source-pull.timer

Deployment

# Auf der GPU-Box, in /srv/mana/:
docker compose -f docker-compose.gpu-box.yml up -d
docker compose -f docker-compose.gpu-box.yml ps

Repo-File ist Source-of-Truth; Live-Datei in /srv/mana/ muss synchron gehalten werden. Aktuell manuell — kein CD-Flow für die GPU-Box.

Cloudflare-Tunnel

Token-managed Windows-Service Cloudflared mit Tunnel mana-gpu-server (UUID 83454e8e-d7f5-4954-b2cb-0307c2dba7a6). Ingress-Konfiguration via API + Cloudflare-Dashboard, NICHT in config.yml. Aktuelle Public-Hostnames: gpu-stt, gpu-tts, gpu-llm, gpu-img, gpu-video, gpu-ollama (alle für die nativen AI-Scheduled-Tasks) plus grafana, git, stats, glitchtip, status (alles *.mana.how, für die Phase-2-Container hier).

Aktive Public-Hostnames (Stand 2026-05-07, config v26):

Hostname Service Zweck
gpu-stt.mana.how :3020 Whisper STT (Scheduled-Task)
gpu-tts.mana.how :3022 Piper TTS
gpu-llm.mana.how :3025 LLM Gateway
gpu-img.mana.how :3023 FLUX image-gen
gpu-video.mana.how :3026 Video-gen
gpu-ollama.mana.how :11434 Ollama API
grafana.mana.how :8000 Phase 2a
git.mana.how :3041 Forgejo (Phase 2b)
stats.mana.how :8010 Umami (Phase 2b)
glitchtip.mana.how :8020 Glitchtip (Phase 2d)
status.mana.how :8090 Status-Page (Phase 2e)
photon.mana.how :2322 Photon Geocoder (cross-LAN-Workaround für mana-geocoding's Probe + privacy-local Provider)

API-Update (idempotent):

# ssh mana-server, dann python3 mit dem cf-update-Pattern aus
# docs/PLAN_OPTION_C.md §4

Deps für die Box

  • WSL2 mit Ubuntu 24.04
  • Docker Engine in WSL2 mit systemd=true
  • .wslconfig mit memory=24GB processors=12 vmIdleTimeout=-1 networkingMode=mirrored
  • Scheduled Task MANA_WSL_Keepalive hält die WSL-VM via wsl --exec sleep infinity warm
  • Systemd Timer mana-source-pull.timer git-pulled /srv/mana/source/ stündlich
  • Windows-Firewall offen für Ports 3100, 9090, 9091 (von Mini-side accessible)
  • netsh interface portproxy zwischen Windows-LAN-IP und WSL2-localhost für die obigen Ports

Alles dokumentiert in docs/WINDOWS_GPU_SERVER_SETUP.md.