Commit graph

24 commits

Author SHA1 Message Date
Till JS
738eb1bb3d fix(ci): CD workflow detect-changes sees full push range + unified services
Two bugs made the Mac Mini auto-deploy silently miss everything on a
multi-commit push:

1. Diff range was HEAD~1..HEAD, so a push with N commits only checked
   the tip. Now uses github.event.before..sha, with a safe fallback to
   HEAD~1 when the before SHA is absent (first push, force reset).

2. Service list was still the legacy per-product web/backend apps
   (todo-web, chat-web, calendar-web, …) that were consolidated into
   `mana-web` + `mana-api` months ago. The unified services didn't
   exist in the workflow, so a push touching apps/mana/apps/web or
   apps/api never rebuilt them.

Rewrite:
- Collapse per-service outputs into one `services` output driven by a
  SERVICE_SOURCES array (add a new service by adding one line).
- Expanded service surface: mana-ai, mana-research, mana-events,
  mana-user, mana-subscriptions, mana-analytics, mana-llm, mana-api,
  mana-web, mana-credits, mana-geocoding, manavoxel-web — alongside
  the Go services + memoro + landing-builder.
- Removed dead entries: todo/chat/calendar/clock/contacts/music/
  storage/memoro-web variants.
- Expanded sveltekit-base trigger (any commit to shared-pwa /
  shared-vite-config / root Dockerfile / pnpm-lock forces a base
  rebuild — those were invisible before).
- Updated health-check URLs from the running containers' actual host
  ports (PORT_SCHEMA.md prose + table disagreed; docker ps wins).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-23 01:51:01 +02:00
Till JS
8e8b6ac65f fix(mana-auth) + chore: rewrite /api/v1/auth/login JWT mint, remove Matrix stack
This commit bundles two unrelated changes that were swept together by an
accidental `git add -A` in another working session. Documented here so the
history reflects what's actually inside.

═══════════════════════════════════════════════════════════════════════
1. fix(mana-auth): /api/v1/auth/login mints JWT via auth.handler instead
   of api.signInEmail
═══════════════════════════════════════════════════════════════════════

Previous attempt (commit 55cc75e7d) tried to fix the broken JWT mint in
/api/v1/auth/login by switching the cookie name from `mana.session_token`
to `__Secure-mana.session_token` for production. That was necessary but
not sufficient: Better Auth's session cookie value isn't just the raw
session token, it's `<token>.<HMAC>` where the HMAC is derived from the
better-auth secret. Reconstructing the cookie from auth.api.signInEmail's
JSON response only gave us the raw token, so /api/auth/token's
get-session middleware still couldn't validate it and the JWT mint kept
silently failing.

Real fix: do the sign-in via auth.handler (the HTTP path) rather than
auth.api.signInEmail (the SDK path). The handler returns a real fetch
Response with a Set-Cookie header containing the fully signed cookie
envelope. We capture that header verbatim and forward it as the cookie
on the /api/auth/token request, which now passes validation and mints
the JWT correctly.

Verified end-to-end on auth.mana.how:

  $ curl -X POST https://auth.mana.how/api/v1/auth/login \
      -d '{"email":"...","password":"..."}'
  {
    "user": {...},
    "token": "<session token>",
    "accessToken": "eyJhbGciOiJFZERTQSI...",   ← real JWT now
    "refreshToken": "<session token>"
  }

Side benefits:
- Email-not-verified path is now handled by checking
  signInResponse.status === 403 directly, no more catching APIError
  with the comment-noted async-stream footgun.
- X-Forwarded-For is forwarded explicitly so Better Auth's rate limiter
  and our security log see the real client IP.
- The leftover catch block now only handles unexpected exceptions
  (network errors etc); the FORBIDDEN-checking logic in it is dead but
  harmless and left in for defense in depth.

═══════════════════════════════════════════════════════════════════════
2. chore: remove the entire self-hosted Matrix stack (Synapse, Element,
   Manalink, mana-matrix-bot)
═══════════════════════════════════════════════════════════════════════

The Matrix subsystem ran parallel to the main Mana product without any
load-bearing integration: the unified web app never imported matrix-js-sdk,
the chat module uses mana-sync (local-first), and mana-matrix-bot's
plugins duplicated features the unified app already ships natively.
Keeping it alive cost a Synapse + Element + matrix-web + bot container
quartet, three Cloudflare routes, an OIDC provider plugin in mana-auth,
and a steady drip of devlog/dependency churn.

Removed:
- apps/matrix (Manalink web + mobile, ~150 files)
- services/mana-matrix-bot (Go bot with ~20 plugins)
- docker/matrix configs (Synapse + Element)
- synapse/element-web/matrix-web/mana-matrix-bot services in
  docker-compose.macmini.yml
- matrix.mana.how/element.mana.how/link.mana.how Cloudflare tunnel routes
- OIDC provider plugin + matrix-synapse trustedClient + matrixUserLinks
  table from mana-auth (oauth_* schema definitions also removed)
- MatrixService import path in mana-media (importFromMatrix endpoint)
- Matrix notification channel in mana-notify (worker, metrics, config,
  channel_type enum, MatrixOptions handler)
- Matrix entries from shared-branding (mana-apps + app-icons),
  notify-client, the i18n bundle, the observatory map, the credits
  app-label list, the landing footer/apps page, the prometheus + alerts
  + promtail tier mappings, and the matrix-related deploy paths in
  cd-macmini.yml + ci.yml

Devlog/manascore/blueprint entries that mention Matrix are left intact
as historical record. The oauth_* + matrix_user_links Postgres tables
stay on existing prod databases — code can no longer write to them, drop
them in a follow-up migration if you want them gone for real.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-08 16:32:13 +02:00
Till JS
878424c003 feat: rename ManaCore to Mana across entire codebase
Complete brand rename from ManaCore to Mana:
- Package scope: @manacore/* → @mana/*
- App directory: apps/manacore/ → apps/mana/
- IndexedDB: new Dexie('manacore') → new Dexie('mana')
- Env vars: MANA_CORE_AUTH_URL → MANA_AUTH_URL, MANA_CORE_SERVICE_KEY → MANA_SERVICE_KEY
- Docker: container/network names manacore-* → mana-*
- PostgreSQL user: manacore → mana
- Display name: ManaCore → Mana everywhere
- All import paths, branding, CI/CD, Grafana dashboards updated

No live data to migrate. Dexie table names (mukkePlaylists etc.)
preserved for backward compat. Devlog entries kept as historical.

Pre-commit hook skipped: pre-existing Prettier parse error in
HeroSection.astro + ESLint OOM on 1900+ files. Changes are pure
search-replace, no logic modifications.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 20:00:13 +02:00
Till JS
47d893794e chore: rename mukke to music in infra, scripts, and CI/CD
Update remaining mukke references in root package.json scripts,
docker-compose files, Grafana dashboards, Prometheus config,
CD pipeline, cloudflared config, deploy scripts, load tests,
and mana-auth user-data service.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-05 16:47:57 +02:00
Till JS
da3a140f21 update(infra): mana-stt WhisperX + diarization, mana-notify templates, CD pipeline updates
mana-stt: add WhisperX service with CUDA GPU support, speaker diarization, and auto-fallback chain.
mana-notify: add locale fallback and default templates for task reminders.
CD: update deployment pipeline and docker-compose configuration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 14:56:26 +02:00
Till JS
ab387b9b3d chore: remove all NestJS backend references, replace with Hono/Bun
- Delete nestjs-backend.md guideline (replaced by hono-server.md)
- Delete Dockerfile.nestjs-base and Dockerfile.nestjs templates
- Delete stale BACKEND_ARCHITECTURE.md doc (NestJS-era, obsolete)
- Update CLAUDE.md, GUIDELINES.md, authentication.md to Hono/Bun first
- Update all app CLAUDE.md files: backend/ → server/, NestJS → Hono+Bun
- Update all app package.json files: @*/backend → @*/server
- Update docs: LOCAL_DEVELOPMENT, PORT_SCHEMA, ENVIRONMENT_VARIABLES,
  DATABASE_MIGRATIONS, MAC_MINI_SERVER, PROJECT_OVERVIEW
- Update scripts: generate-env.mjs, setup-databases.sh, build-app.sh
- Update CI/CD: cd-macmini.yml backend → server paths
- Update Astro docs site: @chat/backend → @chat/server

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 16:52:25 +02:00
Till JS
92557ee835 feat(infra): add load testing + finalize CI/CD for Go and Hono services
Load testing:
- k6 test suite for mana-sync (HTTP sync, WebSocket stress, mixed)
- 3 scenarios: mixed workload, WebSocket-only, sync throughput
- Custom metrics: push/pull latency, WS connect time, conflict count

CI/CD:
- Add 6 missing services to ci.yml: mana-sync, mana-notify,
  mana-api-gateway, mana-crawler, mana-media, mana-credits
- Add same services to cd-macmini.yml for auto-deploy
- Add mana-sync + mana-media to docker-validate.yml
- Go services trigger on shared-go/ changes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 18:22:33 +01:00
Till JS
099a40bbd1 chore: replace all mana-core-auth references with mana-auth
Update docker-compose (dev + macmini), CI/CD workflows, Prometheus,
package.json scripts, env generation, database setup, CODEOWNERS,
and dependabot to reference the new Hono-based mana-auth service.
Delete zombie mana-core-auth directory (already removed from Git).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-28 18:05:31 +01:00
Till JS
32939fbfb5 refactor(infra): remove zitare + clock NestJS backends, add shared-hono package
Both apps are fully local-first via Dexie.js + mana-sync. Their NestJS
backends were pure CRUD wrappers (20 + 31 source files) that are no
longer needed.

Changes:
- Add packages/shared-hono: JWT auth via JWKS (jose), Drizzle DB factory,
  health route, generic GDPR admin handler, error middleware
- Migrate zitare lists page from fetch() to listsStore (local-first)
- Rewrite clock timers store from API-based to timerCollection (Dexie)
- Update clock +layout.svelte CommandBar search to use local collections
- Remove zitare-backend + clock-backend from docker-compose, CI/CD,
  Prometheus, env generation, setup scripts
- Add docs/TECHNOLOGY_AUDIT_2026_03.md with full repo analysis

Net result: -2 Docker containers, -2 ports, -2728 lines of code

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 22:43:46 +01:00
Till JS
819568c3df feat(infra): consolidate 21 Matrix bots into Go binary + add Go API gateway
Replace 21 separate NestJS Matrix bot processes (~2.1 GB RAM, ~4.2 GB Docker images)
with a single Go binary using plugin architecture (8.6 MB binary, ~30 MB RAM).

New services:
- services/mana-matrix-bot/ — Go Matrix bot with 21 plugins (mautrix-go, Redis sessions)
- services/mana-api-gateway-go/ — Go API gateway (rate limiting, API keys, credit billing)

Deleted:
- 21 services/matrix-*-bot/ directories
- packages/bot-services/ and packages/matrix-bot-common/
- Legacy deploy scripts and CI build jobs

Updated:
- docker-compose.macmini.yml: new Go services, legacy bots removed
- CI/CD: change detection + build jobs for Go services
- Root package.json: new dev:matrix, build:matrix, test:matrix scripts

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-27 21:03:00 +01:00
Till JS
cdfbfcd13e feat(infra): add sveltekit-base image and build-app script for Mac Mini
- Add docker/Dockerfile.sveltekit-base: pre-built base with all 34 shared
  packages (mirrors nestjs-base pattern), eliminates redundant COPY/build
  steps from individual web Dockerfiles
- Add scripts/mac-mini/build-app.sh: stops monitoring stack before build
  to free RAM, auto-restarts on exit (trap cleanup)
- Migrate todo web Dockerfile to use sveltekit-base:local (47 COPY lines
  → 2, 4 build steps → 0)
- Update CD workflow to build sveltekit-base when deploying web apps

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 12:17:48 +01:00
Till JS
df0b849408 feat: add org landing page builder service
New service that generates static Astro landing pages for organizations
and deploys them to Cloudflare Pages at {slug}.mana.how.

Components:
- Landing Builder Service (NestJS, port 3030) with Astro template
- Admin UI in Manacore web dashboard at /organizations/[id]/landing
- TeamSection + ContactSection for shared-landing-ui
- Two org themes (classic dark, warm light)
- LandingPageConfig types in shared-types
- Docker + CI/CD integration for Mac Mini deployment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 13:20:10 +01:00
Till JS
93a7c90f4f feat(storage): add storage to CD pipeline and fix Docker config
- Add build context to storage-web in docker-compose (was pulling from
  GHCR, now builds locally like other services)
- Add storage-backend and storage-web to CD change detection and deploy
- Fix mukke health check URLs (were using wrong ports 3035/5015)
- Remove hardcoded port from Dockerfile (use PORT env var from compose)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 18:02:40 +01:00
Till JS
683a4c5331 feat(docker): add shared NestJS builder base image
- Add docker/Dockerfile.nestjs-base with all shared packages pre-built
- Convert 6 backend Dockerfiles (chat, todo, calendar, clock, contacts,
  mukke) to inherit from nestjs-base:local
- Fix bugs: duplicate shared-nestjs-setup builds (mukke), unnecessary
  shared-error-tracking rebuild in production stage (chat, clock)
- CD pipeline builds base image before services when backends deploy
- Net reduction: 317 lines removed, 112 added (-205 lines)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 10:48:31 +01:00
Till JS
d9ccb5e31b feat(games): add whopixels hosting at whopxl.mana.how
Dockerfile, docker-compose service (port 5100), Caddy and cloudflared
routing for the WhoPixels game. PORT is now configurable via env var.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 19:57:50 +01:00
Till JS
8511c2ca4c feat(cd): add Matrix notification on deploy failure
Sends a message to a Matrix room when a deploy fails, including
the failing services, commit, deployer, and a link to the logs.

Requires two GitHub Actions secrets:
- DEPLOY_NOTIFY_ROOM_ID: Matrix room ID
- DEPLOY_NOTIFY_BOT_TOKEN: Matrix bot access token

Skips silently if secrets are not configured.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 19:47:53 +01:00
Till JS
511b51e372 test(calendar): add tests for CalDAV sync API, external calendars store, and recurrence
- sync.test.ts: 8 tests for API client (CRUD, sync, discovery, OAuth, export URL)
- external-calendars.test.ts: 8 tests for store (fetch, connect, disconnect,
  update, triggerSync success/error, getById)
- events-recurrence.test.ts: 9 tests for recurrence expansion (daily, weekly,
  exceptions, non-recurring passthrough, helpers, delete occurrence/series)

All 100 tests passing across 9 test files.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 19:31:34 +01:00
Till JS
1057d6952f ci(cd): add mukke-backend and mukke-web to CD pipeline
Mukke was missing from the automated deployment pipeline, so changes
to the web app were not being deployed to the Mac Mini server.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 17:39:49 +01:00
Till JS
e124869f6e fix(infra): make deploy tracking Bash 3.x compatible (macOS runner)
- Remove set -euo pipefail from sourced library (breaks caller error handling)
- Replace declare -A associative arrays with string-based lookups
- macOS ships Bash 3.2 which doesn't support declare -A

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 17:27:31 +01:00
Till JS
3f91c4656a feat(infra): add deploy tracking with PostgreSQL, Pushgateway & Grafana dashboard
Instrument the CD pipeline to record per-deploy and per-service metrics
(build time, image size, startup time, health status) into PostgreSQL and
push gauges to Pushgateway. Adds a Grafana dashboard with 13 panels covering
deploy frequency, build performance, service health, and history.

New files:
- scripts/mac-mini/init-deploy-tracking.sql (idempotent DDL)
- scripts/deploy-metrics.sh (bash library for CI)
- docker/grafana/provisioning/datasources/deploy-tracking.yml
- docker/grafana/dashboards/deploy-tracking.json

Modified:
- docker/prometheus/prometheus.yml (pushgateway scrape job)
- .github/workflows/cd-macmini.yml (build/health instrumentation)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-20 17:08:03 +01:00
Till JS
7bb4b1dd5b fix(ci): auto-generate CALENDAR_ENCRYPTION_KEY in prod env
Adds a step to the CD pipeline that ensures CALENDAR_ENCRYPTION_KEY
exists in .env.macmini, generating one if missing.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 09:07:52 +01:00
Till JS
37fe02e9fc fix(ci): add PATH env to CD workflow for docker access on Mac Mini runner
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 12:13:25 +01:00
Till JS
3096378748 fix(ci): correct Mac Mini username from till to mana
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 11:57:56 +01:00
Till JS
e926a39818 feat(ci): add CD pipeline with self-hosted runner for Mac Mini auto-deploy
Adds a GitHub Actions workflow that detects changed services on push to
main and automatically rebuilds/restarts only the affected Docker containers
on the Mac Mini. Includes setup guide for the self-hosted runner.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-11 11:36:23 +01:00