managarten/scripts/mac-mini
Till JS 77b2d1eb32 chore(infra): smarter tunnel rebuild — apex via API + sane probes
Two improvements to scripts/mac-mini/rebuild-tunnel.sh based on what
the first prod run actually surfaced.

═══ 1. Apex domain auto-fix via Cloudflare API ═══

`cloudflared tunnel route dns` cannot route the apex of a zone
(error code 1003: "An A, AAAA, or CNAME record with that host already
exists"). The CLI has no command to delete those records. The first
rebuild left mana.how returning 530 because the script silently
failed to route it and we had to fix the apex manually in the
dashboard.

The new `apex_route_via_api()` helper:
  - Detects apex hostnames by dot count (one dot → two-label name)
  - Uses $CLOUDFLARE_API_TOKEN if available
  - Resolves the zone id by name
  - Deletes any existing A / AAAA / CNAME records on the apex
  - Creates a fresh proxied CNAME pointing at <tunnel>.cfargotunnel.com
  - Cloudflare's CNAME flattening at the apex makes this work
    transparently

If $CLOUDFLARE_API_TOKEN is not set, the script logs a warning at the
top of step 6 and falls back to the old behavior (route fails, user
fixes the apex manually). The token needs Zone:DNS:Edit on the
target zone.

═══ 2. Smarter HTTP verification ═══

The first run reported "5 hosts down (404/000)" but those were all
backend services without a root handler — credits/media/llm/mana-api
all return 404 at `/` and 200 at `/health`. The verify pass was
flagging healthy services as down and made the rebuild look more
broken than it was.

New `probe_host()` tries `/health` first, falls back to `/` only if
/health returned 4xx, and prefers a 2xx/3xx root response over a 4xx
/health. `probe_is_down()` only counts 5xx and 000 (libcurl error)
as failures — anything in 1xx-4xx means the request reached the
origin and the tunnel routing is correct, which is the actual thing
the verify pass cares about. `probe_label()` adds a one-word health
summary so the verify log reads "200 ok" / "401 auth required" /
"404 routed (no handler)" / "530 tunnel error" instead of just bare
status codes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-09 17:52:40 +02:00
..
launchd chore(mac-mini): remove all AI service infrastructure (moved to Windows GPU) 2026-04-08 13:06:40 +02:00
backup-databases.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
bootstrap.sh chore: tidy root files + reorganize a few stale docs 2026-04-08 12:15:27 +02:00
build-app.sh fix(macmini): auto-rebuild stale sveltekit-base before per-app web builds 2026-04-08 22:35:46 +02:00
build-landings.sh refactor: rename ManaDeck to Cards across entire monorepo 2026-04-01 11:45:21 +02:00
check-disk-space.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
configure-ollama.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
deploy-v2.sh fix(mana-auth) + chore: rewrite /api/v1/auth/login JWT mint, remove Matrix stack 2026-04-08 16:32:13 +02:00
deploy.sh fix(macmini): mount prometheus config directly so /-/reload picks up edits 2026-04-08 17:25:48 +02:00
ensure-containers-running.sh fix(macmini): repair container auto-recovery (broken --env-file path) 2026-04-08 18:17:31 +02:00
health-check.sh chore(macmini/scripts): runbook hardening — status diff + ingress walk 2026-04-07 22:31:53 +02:00
init-deploy-tracking.sql feat(infra): add deploy tracking with PostgreSQL, Pushgateway & Grafana dashboard 2026-03-20 17:08:03 +01:00
memory-baseline.sh fix(mana-auth) + chore: rewrite /api/v1/auth/login JWT mint, remove Matrix stack 2026-04-08 16:32:13 +02:00
migrate-to-colima.sh fix(mana-auth) + chore: rewrite /api/v1/auth/login JWT mint, remove Matrix stack 2026-04-08 16:32:13 +02:00
move-colima-to-external-ssd.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
notifications.env.example chore: complete ManaCore → Mana rename (docs, go modules, plists, images) 2026-04-07 12:26:10 +02:00
push-schemas.sh feat(news): backend ingester service + curated feed API 2026-04-09 15:53:26 +02:00
README.md chore(mac-mini): remove all AI service infrastructure (moved to Windows GPU) 2026-04-08 13:06:40 +02:00
rebuild-tunnel.sh chore(infra): smarter tunnel rebuild — apex via API + sane probes 2026-04-09 17:52:40 +02:00
restart.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
setup-autostart.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
setup-cloudflared-service.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
setup-docker-logging.sh feat(mac-mini): add stability improvements 2026-02-12 13:33:44 +01:00
setup-forgejo.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
setup-notifications.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
setup-ssh-client.sh feat: add SSH access via Cloudflare Tunnel 2026-01-22 19:27:39 +01:00
setup-umami-db.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
startup.sh fix(mana-auth) + chore: rewrite /api/v1/auth/login JWT mint, remove Matrix stack 2026-04-08 16:32:13 +02:00
status.sh chore: cleanup leftover dirs from ManaCore→Mana rename + document apps/api 2026-04-08 12:12:02 +02:00
stop.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00
sync-tunnel-config.sh chore(infra): make cloudflared-config.yml the single source of truth 2026-04-09 16:37:21 +02:00
tune-tcp.sh feat(skilltree): add achievement system with 26 achievements + monetization report 2026-03-24 12:17:43 +01:00
weekly-report.sh feat: rename ManaCore to Mana across entire codebase 2026-04-05 20:00:13 +02:00

Mac Mini Server Scripts

Scripts for managing the Mana production environment on Mac Mini.

Quick Start (After System Update)

# 1. SSH into Mac Mini (from your local machine)
ssh mac-mini

# 2. Navigate to project
cd ~/projects/mana-monorepo

# 3. Setup auto-start (only needed once)
./scripts/mac-mini/setup-autostart.sh

# 4. Check status
./scripts/mac-mini/status.sh

Scripts Overview

Script Purpose
setup-autostart.sh Configure automatic startup on boot (run once)
startup.sh Main startup script (called by launchd)
health-check.sh Check all services health
status.sh Show full system status
restart.sh Restart all Docker containers
stop.sh Stop all Docker containers
deploy.sh Pull latest images and deploy

First-Time Setup

1. Prerequisites on Mac Mini

# Install Homebrew
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# Install required tools
brew install cloudflared git docker

# Install Docker Desktop
# Download from: https://www.docker.com/products/docker-desktop/

2. Clone Repository

mkdir -p ~/projects
cd ~/projects
git clone https://github.com/Memo-2023/mana-monorepo.git
cd mana-monorepo

3. Configure Cloudflare Tunnel

# Login to Cloudflare
cloudflared tunnel login

# The tunnel is already created (ID: bb0ea86d-8253-4a54-838b-107bb7945be9)
# Credentials should be at: ~/.cloudflared/bb0ea86d-8253-4a54-838b-107bb7945be9.json

4. Configure Environment

# Copy and edit the environment file
cp .env.macmini.example .env.macmini
nano .env.macmini

5. Enable Auto-Start

# This sets up all launchd services
./scripts/mac-mini/setup-autostart.sh

6. Configure Docker Desktop

Open Docker Desktop and enable:

  • Settings > General > Start Docker Desktop when you sign in

Daily Operations

Check Status

./scripts/mac-mini/status.sh

Run Health Check

./scripts/mac-mini/health-check.sh

Restart Services

# Normal restart
./scripts/mac-mini/restart.sh

# Pull latest images and restart
./scripts/mac-mini/restart.sh --pull

# Force recreate containers
./scripts/mac-mini/restart.sh --force

View Logs

# Startup log
tail -f /tmp/mana-startup.log

# Health check log
tail -f /tmp/mana-health.log

# Cloudflare tunnel log
tail -f /tmp/cloudflared.log

# Specific container logs
docker logs -f mana-auth
docker logs -f chat-backend

Stop Services

./scripts/mac-mini/stop.sh

LaunchD Services

Three services are configured to run automatically:

Service Label Purpose
Cloudflared com.cloudflare.cloudflared Tunnel to Cloudflare
Docker Startup com.mana.docker-startup Start containers on boot
Health Check com.mana.health-check Check every 5 minutes
STT Service com.mana.stt Speech-to-Text (Whisper + Voxtral)

Manual Service Control

# Check status
launchctl list | grep -E 'cloudflare|mana'

# Restart a service
launchctl kickstart -k gui/$(id -u)/com.mana.docker-startup

# Stop a service
launchctl unload ~/Library/LaunchAgents/com.mana.docker-startup.plist

# Start a service
launchctl load ~/Library/LaunchAgents/com.mana.docker-startup.plist

Troubleshooting

Docker not starting

# Check if Docker Desktop is running
docker info

# Start Docker Desktop manually
open -a Docker

Cloudflare tunnel not connecting

# Check cloudflared status
pgrep -x cloudflared

# View tunnel logs
tail -50 /tmp/cloudflared.log

# Restart tunnel
launchctl kickstart -k gui/$(id -u)/com.cloudflare.cloudflared

Container health check failing

# Check specific container
docker logs <container-name>

# Restart specific container
docker restart <container-name>

# Check database connectivity
docker exec mana-postgres pg_isready -U postgres

Services not starting on boot

# Re-run setup
./scripts/mac-mini/setup-autostart.sh

# Check launchd errors
launchctl error <exit-code>

# Verify plist files
plutil ~/Library/LaunchAgents/com.mana.*.plist

Push Notifications (Optional)

To receive notifications when health checks fail:

  1. Create a topic at ntfy.sh
  2. Add to your shell profile:
    export NTFY_TOPIC=your-topic-name
    
  3. Subscribe on your phone using the ntfy app

URLs

Once running, services are available at:

Service URL
Unified App https://mana.how
Auth API https://auth.mana.how
API Gateway https://api.mana.how
Forgejo (Git) https://git.mana.how
Grafana https://grafana.mana.how
Status Page https://status.mana.how
GlitchTip https://glitchtip.mana.how
Umami https://stats.mana.how
SSH ssh mac-mini (via cloudflared)

Native Services (non-Docker)

Ollama (LLM)

Ollama runs natively on Mac Mini for LLM inference:

# Check status
curl http://localhost:11434/api/tags

# List models
ollama list

# Pull a model
ollama pull gemma3:4b

AI Services (STT, TTS, LLM, Image-Gen, Video-Gen)

These have moved off the Mac Mini entirely. They run on the Windows GPU server (mana-server-gpu) as Windows Scheduled Tasks. See docs/WINDOWS_GPU_SERVER_SETUP.md for setup, and the per-service services/mana-{stt,tts,llm,image-gen,video-gen}/CLAUDE.md files for endpoint details.

Public URLs (proxied via Cloudflare Tunnel + the Mac Mini gpu-proxy):

  • https://gpu-stt.mana.how
  • https://gpu-tts.mana.how
  • https://gpu-llm.mana.how
  • https://gpu-img.mana.how
  • https://gpu-video.mana.how