managarten/docs/runbooks/photon-on-mana-gpu.md

Runbook — Self-host Photon on mana-gpu

Goal: install WSL2 + Docker Desktop on the Windows GPU server, run a self-hosted Photon (Europe-wide), and cut the geocoding wrapper over to it as the primary privacy: 'local' provider.

Estimated time: 3–4 h, ~1 h of which is unattended download/unpack.

Prerequisites:

• Physical access to mana-gpu (Hyper-V install requires reboot)
• Admin account
• 100 GB free on a fast SSD (current state: 739 GB free on C:, ✓)
• Decent download speed (Europe tarball is ~30 GB)

Pre-checked state (2026-04-28):

• Windows 11 Pro Build 26200 — fully WSL2-ready, no upgrade needed
• 64 GB RAM, 56 GB free
• WSL2: not installed
• Docker Desktop: not installed
• No native CUDA / Ollama / STT / TTS Windows services detected (so Hyper-V activation should not disrupt anything currently running)
• LAN-only — Cloudflare tunnel terminates on the Mac mini, the GPU server is only reachable from the home LAN

---

Phase 1 — WSL2 + Docker Desktop (~1.5 h, requires physical access)

1.1 Enable Windows features (5 min, requires reboot)

Open PowerShell as Administrator and run:

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Then reboot:

Restart-Computer

1.2 Install WSL2 + Ubuntu (10 min)

After reboot, in elevated PowerShell:

wsl --install --no-distribution
wsl --set-default-version 2
wsl --install -d Ubuntu-24.04

When the Ubuntu setup prompt appears, create a user (e.g. mana). Verify:

wsl --list --verbose
# Should show Ubuntu-24.04, VERSION 2, STATE Running

1.3 Install Docker Desktop (15 min)

Download installer:

$dl = "$env:TEMP\DockerDesktopInstaller.exe"
Invoke-WebRequest -Uri "https://desktop.docker.com/win/main/amd64/Docker%20Desktop%20Installer.exe" -OutFile $dl
Start-Process -FilePath $dl -ArgumentList "install","--quiet","--accept-license","--backend=wsl-2" -Wait

Then log out and back in (or reboot). Launch Docker Desktop from the Start Menu, accept the license, uncheck "Send usage statistics" if you care.

In Settings → Resources → WSL Integration: ensure Ubuntu-24.04 is enabled.

Verify from PowerShell:

docker --version
docker run --rm hello-world

If docker run works, Phase 1 is done.

1.4 GPU services sanity check (5 min)

If you have any natively-running ML services (Ollama, image-gen, …), verify they still work after the Hyper-V activation. If something broke, the most common fix is to make sure Docker Desktop is running with WSL2 backend (not Hyper-V backend) — they coexist with CUDA pass-through more smoothly.

---

Phase 2 — Photon container (~45 min, ~30 min of which is download)

2.1 Create data directory + download index (30 min)

In PowerShell (does not need to be admin):

mkdir D:\photon-data -Force
cd D:\photon-data

# ~30 GB compressed; takes 5–30 min depending on your link
Invoke-WebRequest `
  -Uri "https://download1.graphhopper.com/public/europe/photon-db-europe-1.0-latest.tar.bz2" `
  -OutFile "photon-db-europe-1.0-latest.tar.bz2"

If you'd rather start with Germany-only (5.8 GB, faster), use:

Invoke-WebRequest `
  -Uri "https://download1.graphhopper.com/public/europe/germany/photon-db-germany-1.0-latest.tar.bz2" `
  -OutFile "photon-db-germany-1.0-latest.tar.bz2"

Unpack:

# Unpacks into D:\photon-data\photon_data\
tar -xjf photon-db-europe-1.0-latest.tar.bz2

Verify the structure — you should see D:\photon-data\photon_data\ with subdirectories.

2.2 Run Photon container (5 min)

docker run -d `
  --name photon `
  --restart unless-stopped `
  -p 2322:2322 `
  -v D:\photon-data\photon_data:/photon/photon_data `
  -e JAVA_OPTS="-Xmx6g -Xms2g" `
  komoot/photon

Heap notes:

• -Xmx6g (6 GB max heap) is comfortable for Europe; bump to 8g if you see GC churn
• -Xms2g (2 GB initial heap) avoids cold-start expansion latency

Check it boots cleanly:

docker logs -f photon
# Wait until you see lines like: "started Photon" / "OpenSearch ready"
# Press Ctrl+C to stop following

Smoke from the GPU server itself:

curl "http://localhost:2322/api?q=Konstanz&limit=2"
# Expect a GeoJSON FeatureCollection with at least one feature

2.3 Open the firewall (2 min)

The Mac mini needs to reach Photon on port 2322 from the LAN:

New-NetFirewallRule `
  -DisplayName "Photon (mana-geocoding)" `
  -Direction Inbound `
  -Protocol TCP `
  -LocalPort 2322 `
  -Action Allow `
  -Profile Private `
  -RemoteAddress 192.168.178.0/24

(-Profile Private because the network is the home LAN, not Public. Adjust if your network categorisation is different — Get-NetConnectionProfile shows it.)

Verify from the Mac mini:

ssh mana-server 'curl -fsS http://192.168.178.11:2322/api?q=Konstanz | head -c 200'

Should return a Photon GeoJSON response.

---

Phase 3 — Wrapper cutover (already prepared in code, ~15 min to deploy)

The wrapper code already supports a photon-self provider — we just need to set the env-var and recreate the container.

3.1 Add to .env.macmini on mana-server

ssh mana-server
nano ~/projects/mana-monorepo/.env.macmini

Add at the bottom:

PHOTON_SELF_API_URL=http://192.168.178.11:2322

Optionally tighten the chain to drop public Nominatim (saving one redundant fallback):

GEOCODING_PROVIDERS=photon-self,photon

(Default order without this env-var is photon-self,pelias,photon,nominatim. pelias is silently skipped because the Pelias container is stopped. Setting GEOCODING_PROVIDERS=photon-self,photon is more explicit but not required.)

3.2 Recreate mana-geocoding container

cd ~/projects/mana-monorepo
docker compose -f docker-compose.macmini.yml --env-file .env.macmini up -d mana-geocoding

3.3 Verify

# Should now list photon-self as healthy
docker exec mana-geocoding bun -e 'fetch("http://127.0.0.1:3018/health/providers").then(r=>r.text()).then(console.log)'

# Sensitive query should now return REAL results (was empty before cutover)
docker exec mana-geocoding bun -e 'fetch("http://127.0.0.1:3018/api/v1/geocode/search?q=Hausarzt+Konstanz&limit=2").then(r=>r.text()).then(t=>console.log(t.substring(0,400)))'

# Normal query should be served by photon-self (provider:"photon-self", no notice field)
docker exec mana-geocoding bun -e 'fetch("http://127.0.0.1:3018/api/v1/geocode/search?q=Konstanz&limit=1").then(r=>r.text()).then(t=>console.log(t.substring(0,400)))'

Look for:

• "healthy":true for photon-self
• "provider":"photon-self" on the result objects
• No "notice":"fallback_used" (the only notice should appear if photon-self is somehow down)

3.4 Tail the logs for 5 minutes

docker logs -f --since 1m mana-geocoding

Watch for:

• [geocoding-chain] photon-self failed warnings (means cross-LAN issue — debug from there)
• Steady stream of normal 200s from real user traffic (no warnings)

---

Phase 4 — Cleanup (10 min)

4.1 Remove the Pelias stack from the Mac mini

cd ~/projects/mana-monorepo/services/mana-geocoding/pelias
docker compose down -v
# `-v` deletes the volumes too — frees ~5 GB disk

If you want to keep Pelias around as a "break glass" option, omit the -v and just leave the stack stopped (which is the current state).

4.2 Update CLAUDE.md

services/mana-geocoding/CLAUDE.md should reflect the new topology:

• Self-hosted Photon on mana-gpu is the primary, privacy: 'local'
• Public Photon stays as last-resort privacy: 'public' fallback
• Pelias is retired

4.3 Update memory if you keep one

The memory/project_*.md notes should reflect: Pelias retired 2026-04-28, Photon migrated to mana-gpu on $DATE.

---

Phase 5 — Weekly maintenance (10 min/week, optional automation)

GraphHopper publishes a fresh tarball every week. Manual refresh:

cd D:\photon-data
docker stop photon
Invoke-WebRequest -Uri "https://download1.graphhopper.com/public/europe/photon-db-europe-1.0-latest.tar.bz2" -OutFile "photon-db-new.tar.bz2"
mv photon_data photon_data.old
tar -xjf photon-db-new.tar.bz2
docker start photon
# After a smoke-test:
rm -r photon_data.old
rm photon-db-new.tar.bz2

For full automation, schedule the above as a Windows Task Scheduler weekly job. Not urgent — the OSM data being a week stale is fine for our use case.

---

Rollback

If something goes wrong at any step, the rollback is unceremonious:

1. Remove the env-var: unset PHOTON_SELF_API_URL in .env.macmini
2. Recreate the container: docker compose ... up -d mana-geocoding
3. The chain falls back to public Photon + Nominatim — same state as right now

The wrapper code is fully backward-compatible: photon-self registration is opt-in via the env-var alone, no schema migrations, no data dependencies.

---

Open questions for you

1. Europe vs Germany only: Europe is 30 GB compressed → 80 GB unpacked, 6–8 GB heap. Germany is 5.8 GB compressed → 15 GB unpacked, 2–4 GB heap. We recommended Europe; both work. Pick based on whether you ever want to geocode Spanish or Italian addresses.
2. Cleanup of public providers from chain: do you want public Photon + Nominatim to stay as last-resort fallbacks (current behavior), or drop them entirely once self-hosted is up? Keeping them costs nothing if self-hosted is healthy.
3. Nightly Photon restart: OpenSearch occasionally fragments its memory. A weekly restart (or a docker restart photon after the data refresh) keeps things tidy. Not urgent.