VictoriaMetrics + vmalert previously copied prometheus.yml/alerts.yml from
/mnt/prometheus-config/ into /etc/prometheus/ at container start. The copy
silently drifted from the host file whenever the container wasn't restarted —
which is exactly what hid the matrix/element removal from status.mana.how
until 2026-04-08, when VM was still actively scraping the deleted targets
because its in-container config snapshot pre-dated the cleanup.
Now both containers mount ./docker/prometheus directly into /etc/prometheus
(resp. /etc/alerts) read-only and point the binary at it, and deploy.sh
issues POST /-/reload to both after each deploy so config edits go live
without a container recreate.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The first prod deploy of voice quick-add (3b41b39a3) silently fell
back for every transcript: title=transcript verbatim, dueDate=null,
priority=null, labels=[]. The endpoint code was reaching the
fallback() path even though mana-llm was healthy and reachable from
inside the mana-web container.
Root cause: SvelteKit's $env/dynamic/private explicitly excludes any
env var that starts with the public prefix (default PUBLIC_). The
parse-task code read
env.MANA_LLM_URL || env.PUBLIC_MANA_LLM_URL || 'http://localhost:3025'
expecting to fall back to PUBLIC_MANA_LLM_URL when MANA_LLM_URL was
unset, but $env/dynamic/private treats PUBLIC_MANA_LLM_URL as if it
didn't exist on the server side. So it always fell through to
http://localhost:3025, which from inside mana-web is nothing,
fetch threw, and coerce returned the fallback shape.
Two fixes:
1. docker-compose.macmini.yml — set MANA_LLM_URL (no prefix) on
mana-web alongside PUBLIC_MANA_LLM_URL. The PUBLIC_ var is still
needed for the browser-side playground and status page; the
private one is what the parse endpoints actually read.
2. parse-task and parse-habit — drop the dead env.PUBLIC_MANA_LLM_URL
fallback so the next dev who reads the code doesn't think it'd
ever work. Add a comment explaining the SvelteKit gotcha so the
next person setting up a new env var doesn't repeat this mistake.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The mana-service-llm container had OLLAMA_URL pointed at the GPU box's
LAN address (192.168.178.11:11434). On the Mac Mini host that route
works fine, but from inside any Colima container the entire
192.168.178.0/24 subnet gets synthesized RST — Colima's VM "claims"
the LAN range without being able to route to it, so every connect()
returns "Connection refused" before a packet ever leaves the box.
mana-llm started cleanly, reported the configured upstream as
"unhealthy", served an empty /v1/models list, and every chat
completion failed with "All connection attempts failed". The most
visible downstream effect: voice quick-add (parse-task, parse-habit)
silently degraded to its no-LLM fallback for everyone hitting the
local stack — same shape as a successful response, no error log,
just no enrichment.
The Mac Mini already runs a gpu-proxy LaunchAgent
(com.mana.gpu-proxy, /Users/mana/gpu-proxy.py) that forwards
127.0.0.1:13434 → 192.168.178.11:11434 alongside several other GPU
service ports. Pointing OLLAMA_URL at host.docker.internal:13434 and
adding the host-gateway extra_hosts mapping puts mana-llm on the
already-running rail. Verified end-to-end: from inside the container,
GET http://host.docker.internal:13434/api/tags now returns the full
model list (gemma3:4b, gemma3:12b, gemma3:27b, qwen2.5-coder:14b,
nomic-embed-text).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The matrix subsystem was removed in a prior commit. This commit cleans
up the small leftovers that grep found:
- docker-compose.macmini.yml: dropped the "Matrix Stack" port-range
comment, the "matrix" category from the naming convention, and a
stale watchtower comment about Matrix notifications.
- packages/credits/src/operations.ts: removed AI_BOT_CHAT credit
operation type and its definition. It was the billing entry for "Chat
with AI via Matrix bot" — no callers left.
- services/mana-credits gifts schema + service + validation: removed the
targetMatrixId column / param / Zod field. The corresponding
PostgreSQL column was dropped manually with
`ALTER TABLE gifts.gift_codes DROP COLUMN target_matrix_id` on prod.
- docker/grafana/dashboards/{master,system}-overview.json: removed the
`up{job="synapse"}` panel queries — they would have shown No Data
forever now that Synapse is gone.
Production-side cleanup performed in parallel (not in this commit):
- Stopped + removed mana-matrix-{synapse,element,web,bot} containers
- Removed mana-matrix-bot:local, matrix-web:latest,
matrixdotorg/synapse:latest, vectorim/element-web:latest images (~3 GB)
- Removed mana-matrix-bots-data Docker volume
- Removed /Volumes/ManaData/matrix/ media store (4.3 MB)
- DROP DATABASE matrix; DROP DATABASE synapse; on Postgres
Cosmetic leftovers intentionally untouched:
- Eisenhower matrix in todo (LayoutMode 'matrix') — productivity concept
- ${{ matrix.service }} in .github/workflows — GitHub Actions strategy
- services/mana-media/apps/api/dist/.../matrix/* — stale build output
(not in git, regenerated next mana-media build)
This commit bundles two unrelated changes that were swept together by an
accidental `git add -A` in another working session. Documented here so the
history reflects what's actually inside.
═══════════════════════════════════════════════════════════════════════
1. fix(mana-auth): /api/v1/auth/login mints JWT via auth.handler instead
of api.signInEmail
═══════════════════════════════════════════════════════════════════════
Previous attempt (commit 55cc75e7d) tried to fix the broken JWT mint in
/api/v1/auth/login by switching the cookie name from `mana.session_token`
to `__Secure-mana.session_token` for production. That was necessary but
not sufficient: Better Auth's session cookie value isn't just the raw
session token, it's `<token>.<HMAC>` where the HMAC is derived from the
better-auth secret. Reconstructing the cookie from auth.api.signInEmail's
JSON response only gave us the raw token, so /api/auth/token's
get-session middleware still couldn't validate it and the JWT mint kept
silently failing.
Real fix: do the sign-in via auth.handler (the HTTP path) rather than
auth.api.signInEmail (the SDK path). The handler returns a real fetch
Response with a Set-Cookie header containing the fully signed cookie
envelope. We capture that header verbatim and forward it as the cookie
on the /api/auth/token request, which now passes validation and mints
the JWT correctly.
Verified end-to-end on auth.mana.how:
$ curl -X POST https://auth.mana.how/api/v1/auth/login \
-d '{"email":"...","password":"..."}'
{
"user": {...},
"token": "<session token>",
"accessToken": "eyJhbGciOiJFZERTQSI...", ← real JWT now
"refreshToken": "<session token>"
}
Side benefits:
- Email-not-verified path is now handled by checking
signInResponse.status === 403 directly, no more catching APIError
with the comment-noted async-stream footgun.
- X-Forwarded-For is forwarded explicitly so Better Auth's rate limiter
and our security log see the real client IP.
- The leftover catch block now only handles unexpected exceptions
(network errors etc); the FORBIDDEN-checking logic in it is dead but
harmless and left in for defense in depth.
═══════════════════════════════════════════════════════════════════════
2. chore: remove the entire self-hosted Matrix stack (Synapse, Element,
Manalink, mana-matrix-bot)
═══════════════════════════════════════════════════════════════════════
The Matrix subsystem ran parallel to the main Mana product without any
load-bearing integration: the unified web app never imported matrix-js-sdk,
the chat module uses mana-sync (local-first), and mana-matrix-bot's
plugins duplicated features the unified app already ships natively.
Keeping it alive cost a Synapse + Element + matrix-web + bot container
quartet, three Cloudflare routes, an OIDC provider plugin in mana-auth,
and a steady drip of devlog/dependency churn.
Removed:
- apps/matrix (Manalink web + mobile, ~150 files)
- services/mana-matrix-bot (Go bot with ~20 plugins)
- docker/matrix configs (Synapse + Element)
- synapse/element-web/matrix-web/mana-matrix-bot services in
docker-compose.macmini.yml
- matrix.mana.how/element.mana.how/link.mana.how Cloudflare tunnel routes
- OIDC provider plugin + matrix-synapse trustedClient + matrixUserLinks
table from mana-auth (oauth_* schema definitions also removed)
- MatrixService import path in mana-media (importFromMatrix endpoint)
- Matrix notification channel in mana-notify (worker, metrics, config,
channel_type enum, MatrixOptions handler)
- Matrix entries from shared-branding (mana-apps + app-icons),
notify-client, the i18n bundle, the observatory map, the credits
app-label list, the landing footer/apps page, the prometheus + alerts
+ promtail tier mappings, and the matrix-related deploy paths in
cd-macmini.yml + ci.yml
Devlog/manascore/blueprint entries that mention Matrix are left intact
as historical record. The oauth_* + matrix_user_links Postgres tables
stay on existing prod databases — code can no longer write to them, drop
them in a follow-up migration if you want them gone for real.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mana-auth's config.ts has hard-failed startup since commit e9915428c
(phase 2 encryption vault) when MANA_AUTH_KEK is unset in production.
.env.macmini.example documents the variable, but the docker-compose
service definition for mana-auth never had a corresponding
MANA_AUTH_KEK: ${MANA_AUTH_KEK} line in its environment block, so even
when the variable was set in the host .env, it never reached the
container. Result: every restart since yesterday looped on
"MANA_AUTH_KEK env var is required in production".
Added the env passthrough alongside BETTER_AUTH_SECRET with an inline
comment pointing at the generation command + service CLAUDE.md.
Operator action required on the Mac Mini:
KEK=$(openssl rand -base64 32)
echo "MANA_AUTH_KEK=$KEK" >> .env
./scripts/mac-mini/build-app.sh mana-auth # or compose up -d mana-auth
Then back the value up — it cannot be rotated today without re-wrapping
all existing user vaults (no background re-wrap job yet, kek_id column
on encryption_vaults is reserved for the future migration path).
Docker's embedded DNS resolver (127.0.0.11) forwards to the host
resolver, which on the Mac Mini forwards to the home router's
FRITZ!Box DNS. The router keeps a stale negative cache for hours
after a hostname first fails, so any newly added Cloudflare CNAME
(e.g. the GPU public hostnames recreated via the Cloudflare dashboard
during the 2026-04-07 cleanup) appears as "no such host" to the
blackbox probes for the entire negative-cache TTL — even though the
hostname resolves fine via 1.1.1.1 directly the entire time.
Symptom before the fix:
health-check.sh (uses dig @1.1.1.1) → All services healthy ✅
status.mana.how (via blackbox/VM) → 4 GPU services down ❌
The two views were lying to each other in opposite directions —
the public-facing status page reported four healthy services as
down while the operator runbook reported them as up. Confusing
and exactly the kind of monitoring discrepancy a launch should not
ship with.
Fix: pin the blackbox container to public DNS (Cloudflare + Google)
in compose. Blackbox now resolves directly against 1.1.1.1, bypassing
the home-router negative cache entirely. After the recreate the four
GPU probes flipped from probe_success=0 to probe_success=1 within
one scrape interval, and status.mana.how went from 38/42 to 41/42
(only gpu-video remains down — LTX Video Gen is intentionally not
deployed on the Windows GPU box yet).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three Mac Mini infrastructure follow-ups bundled:
1. docker-compose.macmini.yml — drop ghost backend env vars from
the mana-app-web service (todo, calendar, contacts, chat, storage,
cards, music, nutriphi `PUBLIC_*_API_URL{,_CLIENT}` plus the memoro
server URLs). The matching consumers were removed in the earlier
ghost-API cleanup commits, so these env entries had been wiring
nothing into the running container for several deploys. Force-
recreating mana-app-web after pulling this commit will pick up
the slimmer env automatically.
2. docker-compose.macmini.yml — bump `mana-mon-blackbox` mem_limit
from 32m to 128m. blackbox-exporter v0.25 sits north of 32m
under load and was OOM-restart-looping every ~90 seconds, which
in turn made `status.mana.how` and the prometheus probe metrics
stale (since the scraper was missing every other window).
3. docker/prometheus/prometheus.yml — split `blackbox-gpu` into two
jobs:
- `blackbox-gpu` now probes `/health` via the http_health
module, because the GPU services (whisper STT, FLUX image
gen, Coqui TTS) return 401/404 on `/` by design (auth or
API-only). The previous http_2xx-on-`/` probe was reporting
all four as down even though they answered `/health` with
200, which inflated the down count on status.mana.how.
- `blackbox-gpu-root` keeps the http_2xx-on-`/` probe for
Ollama, which has no `/health` endpoint but does answer
2xx on its root.
Both jobs share the same blackbox-exporter relabel rewrite so
the targets are routed through the exporter container, not
scraped directly by VictoriaMetrics.
Verified post-fix: status.mana.how reports 41/42 services up (only
`gpu-video` remains down — LTX Video Gen is intentionally not
deployed yet on the Windows GPU box).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The unified mana-web container needs MANA_STT_URL + MANA_STT_API_KEY at
runtime so its server-side proxies (/api/v1/memoro/transcribe and
/api/v1/dreams/transcribe) can reach mana-stt with the right credentials.
The browser never holds the key.
URL points at the public tunnel (https://gpu-stt.mana.how → Cloudflare
tunnel mana-gpu-server → Windows GPU box localhost:3020) so the resolver
works regardless of where the container runs. The API key is sourced from
the Mac Mini .env, which is gitignored.
Without this, the proxies short-circuit with HTTP 503 "mana-stt is not
configured" — observed today on first deploy of the recording pipeline.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Five small follow-ups on Phase 1b:
- docker-compose.macmini.yml: add the mana-events container with the
same shape as mana-credits, expose port 3065, add a Traefik route
for events.mana.how, and inject PUBLIC_MANA_EVENTS_URL into the
mana-web container so the SvelteKit SSR + browser both reach it.
- mana-events: background sweeper that deletes rsvp_rate_buckets
rows older than 2h every hour. Without it, long-published events
accumulate one row per traffic-hour forever (FK cascade only fires
on snapshot delete).
- PublicRsvpList: track consecutiveFailures and only show the error
banner after two failures in a row, so a single mid-poll network
hiccup doesn't flash a 30s error the user can't act on.
- apps/mana/apps/web: declare postgres as a devDep (already imported
by the e2e spec via pnpm hoisting, now explicit).
Stalwart requires username without domain for auth and the 'user' role
for SMTP access. Update SMTP_USER from admin to noreply.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The noreply account lacks SMTP auth permissions in Stalwart. Use the
admin account for now — SMTP_FROM still sends as noreply@mana.how.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Umami database was re-initialized with empty website table. Created
new ManaCore Web website in Umami and updated the ID in docker-compose
and .env.development. Fixes stats.mana.how 400 errors.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add SMTP_INSECURE_TLS env var to skip certificate verification for
internal Docker-network SMTP connections. Stalwart's self-signed cert
uses 'localhost' as CN which doesn't match the 'stalwart' hostname.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set SMTP defaults to use internal Stalwart server (stalwart:587) with
noreply@mana.how credentials. Add stalwart as dependency for mana-notify.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Map host 8443 to container 8080 (HTTP admin UI). Use wget for
healthcheck since curl is not available in the Stalwart image.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The old image name stalwartlabs/mail-server doesn't exist on Docker Hub.
The correct image is stalwartlabs/stalwart.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add self-hosted Stalwart mail server (Rust, ~50MB RAM) to replace Brevo
as SMTP provider. mana-notify now sends via stalwart:587 internally.
Ports exposed: 25 (SMTP), 587 (submission), 465 (SMTPS), 993 (IMAPS),
8443 (web admin). Requires DNS setup (MX, SPF, DKIM, DMARC) and router
port-forwarding to complete the migration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
SMTP_USER was empty because it wasn't in .env and had no default.
Add the Brevo account as default (was previously hardcoded in mana-auth).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Copy the volume-mounted generate.sh to /tmp before executing, so a
concurrent git pull doesn't corrupt the file mid-read.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mana-notify was using NOTIFY_SERVICE_KEY (defaulting to dev-service-key)
while mana-auth sends MANA_CORE_SERVICE_KEY. Use the same env var so
mana-auth can authenticate with mana-notify.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace direct Brevo SMTP sending with HTTP calls to mana-notify's
notification API. This centralizes all email configuration in one
service (mana-notify) and removes the nodemailer dependency from
mana-auth. SMTP provider is now swappable via a single env var.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Upgrade shared-logger to dual-mode: JSON lines in production, console
in dev. Adds configureLogger() for service name + request ID.
- Add requestLogger middleware to shared-hono with request ID generation
and structured request/response logging.
- Align Promtail config with new JSON field names (requestId, ts, service).
- Add PUBLIC_GLITCHTIP_DSN + PUBLIC_UMAMI_WEBSITE_ID to mana-web docker config.
- Add /status page that polls all backend /health endpoints server-side.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mirrors the frontend unification (single IndexedDB) on the backend.
All services now use pgSchema() for isolation within one shared database,
enabling cross-schema JOINs, simplified ops, and zero DB setup for new apps.
- Migrate 7 services from pgTable() to pgSchema(): mana-user (usr),
mana-media (media), todo, traces, presi, uload, cards
- Update all DATABASE_URLs in .env.development, docker-compose, configs
- Rewrite init-db scripts for 2 databases + 12 schemas
- Rewrite setup-databases.sh for consolidated architecture
- Update shared-drizzle-config default to mana_platform
- Update CLAUDE.md with new database architecture docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add missing shared-uload package copy and zitare content build step to
Dockerfile. Replace wget/httpx healthchecks with bun fetch and stdlib
urllib to remove external dependencies in containers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mana-stt: add WhisperX service with CUDA GPU support, speaker diarization, and auto-fallback chain.
mana-notify: add locale fallback and default templates for task reminders.
CD: update deployment pipeline and docker-compose configuration.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extract ~120 hardcoded German strings from 14 Svelte components into i18n locale
files using svelte-i18n $t() calls. Add new translation sections (taskForm, filters,
tags, subtasks, durationPicker, kanban, toolbar) across all 5 languages (de/en/fr/es/it).
Also add missing shared common translations for Spanish, French, and Italian
(150+ keys each) in packages/shared-i18n.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Dockerfile using sveltekit-base:local pattern (port 5038)
- docker-compose.macmini.yml entry with Traefik labels for memoro.mana.how
- Delete legacy authService.ts and auth.ts (app uses shared-auth-stores)
- Remove middleware env vars from env.ts and app.d.ts (dead code)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Parse tier data automatically from mana-apps.ts (awk, read-only volume
mount) so the status page stays in sync without manual updates. Shows
founder/alpha/beta/public cards with per-app development status.
Tier data is also included in status.json for ManaScore consumption.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Production build was falling back to localhost:3050 for mana-sync because
PUBLIC_SYNC_SERVER_URL was not set as a build-time ARG. Vite bakes
import.meta.env vars at build time, so the runtime docker-compose
environment section alone is insufficient.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Dockerfile for audio-server (Bun + ffmpeg)
- docker-compose.macmini.yml entries for memoro-server (3015) and memoro-audio-server (3016)
- Dev commands: dev:memoro:server, dev:memoro:audio-server, dev:memoro:app, dev:memoro:full
- MEMORO_* env vars in .env.development
- web: add PUBLIC_MEMORO_SERVER_URL env var to env.ts and .env.example
- web: rewrite transcriptionService → POST /api/v1/memos (new server path)
- web: rewrite spaceService → /api/v1/spaces/* (aligned with actual Hono routes)
- server: fix callAudioServer param name audioPath (was filePath) in memos.ts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All 4 apps now use the same local build pattern as the other 33 apps.
Only umami (external project) keeps its GHCR image.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- inventar-web: second nested <button> in list view also converted to
<div role="button"> to fix Svelte 5 HTML validation
- uload-server: port changed from 3041 to 3070 to avoid conflict with
Forgejo which also binds port 3041
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Forgejo runner has no macOS binary — Docker-based runner can't access
host filesystem/SSH needed for CD. GitHub CD via native self-hosted
runner handles all deployments. Forgejo remains a push-mirror for
backup and visibility.
- Remove .forgejo/workflows/cd-macmini.yml
- Remove forgejo-runner service from docker-compose
- Update mirror workflow comments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Loki healthcheck interval is 5min — using service_healthy blocks Promtail
from starting for up to 5 minutes. service_started is sufficient since
Loki reports /ready immediately after startup.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- docker-compose: fix port mapping 2222:2222 → 2222:22 (sshd listens on 22)
- .forgejo/workflows/cd-macmini.yml: new CD pipeline for Forgejo Actions
(mirrors .github/workflows/cd-macmini.yml, run_url points to git.mana.how)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Now that VirtioFS is fixed (colima /Users/mana mount restored),
file bind-mounts work correctly. Replace the cp-in-entrypoint workaround
with a direct file mount to /etc/alertmanager/alertmanager.yml.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
vmalert: was copying prometheus.yml into /etc/alerts/ causing parse
failure. Now only copies alerts.yml (the actual rules file).
synapse: mana-auth (Better Auth) has no OIDC discovery endpoint,
so disable OIDC and enable password auth until OIDC is implemented.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
YAML '>' block scalar preserves newlines for over-indented lines,
causing 'exec binary' to run without its arguments. Fix: use JSON
array format for all entrypoints to avoid YAML folding entirely.
Also fixes SearXNG entrypoint path: image updated from
dockerfiles/docker-entrypoint.sh to entrypoint.sh.
Affected: victoriametrics, loki, vmalert, alertmanager, searxng, synapse
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
