Commit graph

595 commits

Author SHA1 Message Date
Till JS
177734a860 fix(tsconfig): unblock shared-types consumers
shared-types/src/index.ts re-exports with explicit .ts extensions
(Tailwind v4 module resolver needs them). TS 5.7 requires consumers
to opt in via allowImportingTsExtensions. The flag only type-checks
when noEmit:true; the NestJS builder also needs
rewriteRelativeImportExtensions so tsc still emits valid JS.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 18:53:55 +02:00
Till JS
12be75e6a6 fix(broadcast): track route paths + shared-branding tsconfig
Two fixes surfaced by the end-to-end smoke test.

1. broadcast-track.ts: inner route paths double-prefixed.
   Routes were declared as '/track/open/:token' etc, then
   mounted at '/api/v1/track', yielding '/api/v1/track/track/open/:token'
   — every tracking endpoint returned 404. Dropped the redundant
   '/track/' prefix so the full path is now
   '/api/v1/track/{open,click,unsubscribe}/:token' as the
   orchestrator + client both expect.

   Verified with live curl:
   - /track/open/BAD → 200 image/gif 42 bytes (graceful no-signal)
   - /track/click/?url missing → 400 missing url
   - /track/click?url=javascript: → 400 bad url
   - /track/click?url=https://ok + bad token → 302 graceful
   - /track/unsubscribe/BAD GET → 400 HTML
   - /track/unsubscribe/BAD POST → 400 (RFC 8058)

2. shared-branding/tsconfig.json: allowImportingTsExtensions
   missing. shared-types/src/index.ts uses explicit .ts
   imports (intentional, for Tailwind's module resolver); any
   downstream tsconfig without allowImportingTsExtensions emits
   8 errors. shared-auth already had this fix — shared-branding
   gets the same treatment. noEmit:true is set, so no rewrite
   flag needed.

   Verified: shared-branding pnpm check → 0 errors.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 18:30:47 +02:00
Till JS
1861e89d45 chore(broadcast): wire mana-mail into env pipeline + push schema
The three final pre-dogfood items:

1. drizzle.config: schemaFilter now includes 'broadcast' alongside
   'mail'. Without this, `bun run db:push` skipped the broadcast
   tables — schema existed in code but not in Postgres. Tested via
   db:push + psql \dt (3 tables created: campaigns, events, sends).

2. .env.development: new MANA-MAIL SERVICE section with Stalwart
   knobs + broadcast config (tracking secret, rate limits, send
   throttle). DEV secret is explicitly labelled non-production —
   prod rotates via env.

3. generate-env.mjs: new block writes services/mana-mail/.env on
   `pnpm setup:env`. Mirrors the invoices / research / events
   pattern. All 16 broadcast/mail vars flow through from SSOT.

Verified end-to-end:
- pnpm setup:env → services/mana-mail/.env contains
  BROADCAST_TRACKING_SECRET + rate limits
- bun run src/index.ts → /health returns 200 with the new config
- psql → broadcast.campaigns / events / sends are materialised

Broadcast module is now fully ready to send real mail — nothing
else required before the first dogfood campaign.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 16:21:57 +02:00
Till JS
260dd312a9 feat(broadcast): M8 DNS auth check (SPF / DKIM / DMARC)
Closes the last plan milestone. Users can verify their sending-domain
setup without leaving the broadcast settings page.

Server (mana-mail)
- services/dns-check.ts: parseSpf / parseDkim / parseDmarc are pure
  functions. SPF accepts include:<mailDomain>, flags weak (+all) and
  wrong (include missing) and multi-record (RFC 7208 §3.2). DKIM
  needs v=DKIM1 + a p= public-key segment. DMARC requires v=DMARC1,
  flags p=none as weak (monitoring only), ok on quarantine/reject.
  All three are case-insensitive.
- lookupTxt(): DNS-over-HTTPS against Cloudflare 1.1.1.1 — avoids
  the Bun/container udp-resolver flakiness and works everywhere.
  Multi-string TXT (`"a" "b"`) get concatenated before parsing.
- checkDomain(): one call, three parallel DoH lookups, returns a
  structured result with suggested copy-paste records scoped to the
  user's actual mail domain from config.
- Route: GET /v1/mail/dns-check?domain=&selector= (JWT auth). Zod
  validates the domain looks sensible before hitting DoH.
- 16 unit tests covering all three parsers + multi-record edge case.

Client
- api.ts: runDnsCheck(domain, selector?) helper with typed result.
- components/DnsCheckBanner.svelte: derives domain from the default
  from-email (after @), calls the check on-demand, renders per-record
  status chips (ok / weak / wrong / missing) with messages, exposes
  copy-pasteable SPF + DMARC records when anything's off. DKIM setup
  is provider-specific so we show a hint rather than a canned record.
  Last-check timestamp persists to settings.dnsCheck so the banner
  survives a reload without re-hitting the API.
- Wired into SettingsForm between Impressum and Standard-Footer —
  where the user is already thinking about "what's required to
  actually send".

All checks clean:
- webapp pnpm check: 0 broadcast errors (4 pre-existing articles errors
  from parallel Spaces work, unrelated)
- mana-mail tests: 36/36 across tracking-token + link-rewriter + dns-check
- mana-mail build: 2.51 MB (+8 KB for juice — dns-check itself is ~3 KB)

Plan: docs/plans/broadcast-module.md §M8. All 10 milestones now done.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 15:48:03 +02:00
Till JS
a312d98f09 feat(broadcast): click-link tracking + send throttle
Closes the last two dogfood blockers before real-campaign use.

link-rewriter.ts
- rewriteClickLinks(): walks <a href="http…"> in the HTML body and
  replaces each URL with /api/v1/track/click/{token}?url={original}
  so clicks go through the tracking endpoint. Regex-based because
  Tiptap output is well-formed; returns a count for debugging.
- Leaves mailto: / tel: / anchor fragments alone — wrapping those
  breaks the recipient's native handler and accomplishes nothing.
- `skipUrls` param carries the unsubscribe + web-view URLs (already
  tracking endpoints themselves) so they don't get double-wrapped.
- 11 unit tests covering http/https rewriting, skip list, non-http
  schemes, attribute preservation, multi-link count, quoted-attr
  variants, idempotency.

Orchestrator wiring
- substituteUrls now calls rewriteClickLinks after the preview-
  placeholder swap and before the open-pixel injection. The
  unsubscribe + web-view URLs from this same function are passed
  in as skip entries so they survive the pass untouched.
- Constructor gains `sendThrottleMs` param (default 150ms).
- Main send loop awaits sleep(throttleMs) between iterations. 150ms
  = ~6/sec = ~360/min, safely below most SMTP provider limits.
  100-recipient campaign = ~15s extra wall-clock but that's fine
  for MVP (and most campaigns are way smaller).

Config
- New env BROADCAST_SEND_THROTTLE_MS (default 150). Wired from
  loadConfig to the orchestrator constructor.

The broadcast module is now functionally complete for dogfooding.
Remaining before a real campaign can actually go out: run
`cd services/mana-mail && bun run db:push` to materialise the
broadcast.* schema tables.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 15:07:58 +02:00
Till JS
d887fc125d feat(broadcast): settings + detail view + compliance polish
Closes the "could actually dogfood" gap: legal address can be set,
sent campaigns have a proper view with live stats, and the send path
respects DSGVO.

Webapp
- components/SettingsForm.svelte: sender defaults + Impressum (required,
  highlighted amber until filled) + footer. Matches the invoices
  SenderProfileForm pattern — immediate save, dedicated section per
  concern.
- /broadcasts/settings/+page.svelte: mounts the form. ComposeView step
  3's "Einstellungen öffnen" CTA now lands somewhere.
- views/DetailView.svelte: read-only view for sent/scheduled/cancelled
  campaigns. 5-card stats grid (sent, open, click, bounce, unsub) with
  rate percentages. Polls mana-mail every 30s for up to 30 min after
  mount, persists back to Dexie via applyServerStatus so the list view
  + widget catch up. Includes a preview of the actual rendered campaign
  so "what went out" is visible after the fact.
- /broadcasts/[id]/+page.svelte: DetailView for non-drafts; drafts
  bounce to /edit via $effect-triggered goto.
- ListView row-click now routes by status (draft → edit, else → detail).

mana-mail compliance
- Orchestrator loadUnsubscribedEmails(): queries broadcast.sends WHERE
  status='unsubscribed' scoped to the user, filters the recipient list
  BEFORE any send rows get written. Campaign's totalRecipients reflects
  the post-skip count so open rates aren't inflated by "virtual sends".
  Skipped count surfaces in result.errors for the UI to show.
- jmap-client.submitEmail: new extraHeaders param. Sets custom headers
  via JMAP's `header:<Name>:asText` property convention.
- Orchestrator sets RFC 8058 headers per recipient:
    List-Unsubscribe: <https://.../track/unsubscribe/{token}>
    List-Unsubscribe-Post: List-Unsubscribe=One-Click
  This is what makes Gmail / Apple Mail show their native "Abmelden"
  button in the message header (not just a body link).

All checks clean: 0 TS errors, 37/37 webapp tests, 9/9 tracking-token
tests, mana-mail bun build = 2.50 MB.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 14:43:36 +02:00
Till JS
f17383f9f2 feat(broadcast): M4 bulk-send via mana-mail + tracking infrastructure
End-to-end send path lives: click "Jetzt senden" in step 4 → client
resolves recipients → POST /v1/mail/bulk-send → mana-mail loops through
JMAP with per-recipient signed URLs → status flips draft → sent.

mana-mail (backend)
- New Postgres schema `broadcast.{campaigns,sends,events}` in Drizzle.
  Campaigns + sends keyed on the webapp's local ids so joins are free;
  events append-only with send_id FK, dedup at query-time not write-time
  so tracking pixel hits don't contend on a transaction.
- tracking-token.ts: HMAC-SHA256 over JSON({campaignId, sendId, nonce}),
  base64url.base64url encoded. JSON inner payload instead of delimiter
  splits so IDs can contain any character. timingSafeEqual for the HMAC
  comparison. 9 unit tests covering roundtrip / tamper / malformed.
- broadcast-orchestrator.ts: takes pre-resolved recipient list, inlines
  CSS once via juice (webResources.images=false so no external fetches
  slow the loop), per-recipient substitutes `{{unsubscribe_url}}` /
  `{{web_view_url}}` + injects open pixel, submits each mail through
  the user's own JMAP account. Writes sends rows first (status=queued)
  so a crash mid-loop leaves truthful DB state. Returns aggregate
  stats + per-email errors.
- Routes: POST /v1/mail/bulk-send (JWT, cap at 5000 recipients via
  zod + config), GET /v1/mail/campaigns/:id/events (JWT, aggregates
  opens + clicks + unsubscribes with COUNT DISTINCT for the "unique"
  metric), GET/POST /v1/track/{open,click,unsubscribe}/:token (public,
  no auth, signed URL is the only gate).
- Track routes mounted OUTSIDE /api/v1/mail/* because the JWT
  middleware guards that subtree — recipients aren't logged in.
- Config: BROADCAST_TRACKING_SECRET (separate from SERVICE_KEY so the
  blast radius of a leak stays narrow),
  BROADCAST_MAX_RECIPIENTS_PER_CAMPAIGN (default 5000),
  BROADCAST_MAX_RECIPIENTS_PER_HOUR (default 500, not yet enforced).
- Added juice@^11 dependency.

Webapp (client)
- api.ts: sendCampaign() resolves the audience from Dexie contacts,
  renders the full email HTML + plaintext with placeholders, POSTs to
  mana-mail. Contacts NEVER leave the client decrypted — the server
  only sees the flat recipient list the user's client produced.
- fetchCampaignStats() for M7 dashboard/detail polling.
- ComposeView step 4 replaced: confirmation modal with "sicher?"
  question, sending state with spinner, done state with delivered
  count + expandable per-email error list + "Zur Übersicht" button.
- Status transitions to 'sent' with cached stats after successful
  send via applyServerStatus.

Known M4 gaps (fill in M5)
- Open/click/unsubscribe track endpoints return valid responses but
  event dedup is rough — one insert per hit, dedup at query time
  only. M5 adds windowed IP-hash dedup.
- Synchronous send loop. 100 recipients ≈ 15s blocking. M5/M6 moves
  this to an async job queue with SSE progress.
- Each recipient generates a "Sent" folder entry in the user's
  Stalwart mailbox. Fine for 50-recipient newsletters, silly for
  5000. Phase 2 carves out a dedicated broadcast mailbox.

Plan: docs/plans/broadcast-module.md §M4.
Next: M5 open/click tracking with dedup + rate-limits.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-21 13:53:13 +02:00
Till JS
1d3794f96c feat(mana-ai): Prometheus metrics for tool-calls, loop rounds, provider errors
Three new counters + one histogram fill the observability gap from
the function-calling migration:

- mana_ai_tool_calls_total{tool, policy, outcome} — one tick per
  tool_call the planner produced. `outcome` is `deferred` on the
  server (stub onToolCall records for later client execution);
  webapp runner will emit success/failure once it grows its own
  Prom surface.
- mana_ai_planner_rounds (histogram, buckets 1..5) — distribution of
  rounds consumed per iteration. Runs close to the cap signal a
  planner struggling with the mission objective.
- mana_ai_provider_errors_total{provider, kind} — structured errors
  surfaced from mana-llm. Kind mirrors the ProviderError hierarchy
  added in commit 1 of the migration (blocked/truncated/auth/
  rate_limit/capability/unknown).

Plumbing:
- llm-client.ts parses mana-llm's `{detail: {kind, message}}` 4xx/5xx
  body shape and re-throws as ProviderCallError carrying the kind.
- tick.ts observes metrics at the natural emission points — rounds
  + per-call counter after runPlannerLoop returns, provider_errors
  in the catch block.

Grafana dashboards + status.mana.how already pick up the
collectDefaultMetrics prefix, so these metrics land in the existing
mana-ai panel without scraper changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 20:48:29 +02:00
Till JS
38d35247cd feat(spaces): end-to-end shared-space sync (membership lookup + plaintext)
Closes the gap between "invite flow UI exists" and "two users in the
same space actually see each other's data". Three pieces land together
because they're meaningless without each other.

mana-auth — new internal endpoint:
  GET /api/v1/internal/users/:userId/memberships
  Returns [{organizationId, role}, ...] for the user. mana-sync uses
  this to populate the multi-member RLS session config.

mana-sync — membership lookup:
  new internal/memberships package with an HTTP client + 5 min
  per-user cache, fail-open (empty list = pre-Spaces behavior).
  Config gets MANA_AUTH_URL (default http://localhost:3001).
  Handler.NewHandler takes the Lookup. Every Push/Pull/Stream call
  now passes spaceIDsFor(userID) to Store methods.
  GetChangesSince + GetAllChangesSince extend their WHERE clause:
    WHERE (user_id = $1 OR space_id = ANY($memberSpaces))
  so co-members see each other's rows, not just the author.

apps/web — encryption skip for shared-space records:
  encryptRecord now checks record.spaceId:
    - `_personal:<userId>` sentinel OR no active shared space → encrypt
      with user master key (E2E as today).
    - Active space resolves to non-personal type AND spaceId matches
      that space → skip encryption; write lands plaintext.
  decryptRecord is unchanged because its per-field isEncrypted() guard
  already passes plaintext through.
  Phase-1 compromise: shared-space data is protected by server RLS
  only, not E2E. Phase 2 adds per-Space shared keys with per-member
  wrap — tracked in docs/plans/spaces-foundation.md.

Plus docs/plans/shared-space-smoketest.md: step-by-step Zwei-User-Test
mit erwarteten Ergebnissen und Debugging-Hinweisen bei Problemen.

Build + go test + web check all green.

Plan: docs/plans/spaces-foundation.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 20:46:53 +02:00
Till JS
da373491b8 chore(mana-llm): thread GOOGLE_API_KEY + default model into local compose
Matches the macmini compose — Google Gemini was already wired in the
provider adapter (commit 2 of the function-calling migration) but the
local dev stack's compose never passed the env through, so the
container booted without the provider and every tool-calling request
fell back to Ollama (unreachable in local dev, LAN-only GPU box).

With this in place the local mana-llm healthcheck reports both
`google` and `openrouter` as healthy and the webapp planner hits
Gemini Flash for real.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 20:42:21 +02:00
Till JS
79a6da3e2e feat(spaces): move access tier from user to space
Migration from user-level tier to Space-level tier, following the
Spaces foundation plan. User-visible effect: the tier that gates
module access now belongs to the active Space, not the user account.
Personal Spaces inherit the user's old tier on signup so nothing
downgrades.

shared-types:
- New SpaceTier type ('guest' | 'public' | 'beta' | 'alpha' | 'founder').
- New spaceTierMeets(actual, required) helper.
- SpaceMetadata gains an optional `tier` field.

mana-auth:
- createPersonalSpaceFor reads user.accessTier and stamps it into the
  personal Space's metadata.tier. A founder-tier user setting up their
  first Space keeps founder access in that Space.
- databaseHooks.user.create.after now forwards accessTier into the
  personal-space creator.

apps/web (scope layer):
- ActiveSpace gains a required `tier: SpaceTier`; rawToActiveSpace
  reads it from organization.metadata, defaulting to 'public' if
  missing or invalid.
- New getEffectiveTier(userFallback) helper resolves the tier to use
  for gating: prefers the active Space's tier, falls back to the
  caller-supplied user tier during the boot window.

apps/web ((app) layout):
- `effectiveTier` $derived replaces every authStore.user?.tier reference
  in the layout's access-gating logic (appItems, routeBlocked,
  routeTierLabels). AuthGate deeper in the UI keeps using user.tier as
  its own fallback — the tier move is additive, not destructive.

What this does NOT do yet:
- The user.accessTier column still exists and is still the initial
  source for personal-space tier. Removing it is a later cleanup once
  every code path reads through the Space primitive.
- No admin API for setting tier on a Space (PUT /api/v1/admin/spaces/
  :id/tier). Follow-up when admin tooling needs it — today admins still
  set user.accessTier, which flows to the personal space on next
  signup.

Resolves the MANA_APPS-tier-patch workaround memory: future sessions
can adjust tier per Space instead of per User.

0 errors across 7151 files. 10/10 scope tests pass.

Plan: docs/plans/spaces-foundation.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 20:10:06 +02:00
Till JS
88e3adb9d3 feat(spaces): multi-member RLS policy in mana-sync (forward-compat)
Adds the second RLS policy needed for shared spaces. Users can read
rows in any space they're a member of, in addition to their own rows.

Changes:
- New policy sync_changes_space_member_read (SELECT only) uses
  app.current_user_space_ids session config: rows with space_id in
  that comma-separated list pass RLS.
- WITH CHECK is not extended — writes still require user_id match, so
  only the author can write. Members read, owner/author writes.
- withUser() is now a thin wrapper around withUserAndMemberships(),
  which accepts the caller's Space membership list and sets the new
  session config alongside app.current_user_id.
- The comma-join is empty-filtered so stray blank entries can't match
  rows with literal empty space_id (defense in depth).

Forward-compatible: today every space has exactly one member (the
author), so the membership list is always empty and the new policy
is a no-op — user_id isolation remains the only active guard.

When shared spaces start being used (clubs/teams/brand spaces with
invites), the HTTP handlers will fetch the caller's membership from
mana-auth and pass it to withUserAndMemberships. No migration needed
at that point — the policy is already live.

Subscription fan-out (WS/SSE broadcast to all space members) is still
per-user; that's a follow-up tied to the membership lookup infra.

Go build + existing tests pass.

Plan: docs/plans/spaces-foundation.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 19:55:17 +02:00
Till JS
0d613e1846 feat(ai): thread TokenUsage through runPlannerLoop → mana-ai budget
Carries per-round token counts from the mana-llm response body
(prompt_tokens + completion_tokens) back through LlmCompletionResponse
→ PlannerLoopResult. The loop sums across rounds and exposes a single
aggregate on result.usage.

Lets mana-ai's tick re-activate per-agent daily-token budget tracking
— tokensUsed was stubbed to 0 in the migration commit (6) because the
loop didn't surface usage yet. Now recordTokenUsage + agentTokenUsage24h
get real numbers again, and the mana_ai_tokens_used_total Prometheus
counter is accurate.

Additive only: consumers without usage needs ignore the new field,
and providers that don't return usage produce zeros (not undefined —
the loop still exposes the object so downstream branches stay trivial).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 18:21:34 +02:00
Till JS
e10c2436a6 feat(spaces): thread space_id through mana-sync protocol + storage
Server-side:
- sync_changes gains a nullable space_id TEXT column + partial index
  on (user_id, space_id, app_id, created_at) WHERE space_id IS NOT NULL.
- RecordChange takes spaceID as a first-class parameter; *string so
  empty strings land as real SQL NULL and the partial index skips them.
- ChangeRow + all three SELECTs (GetChangesSince, GetAllChangesSince,
  StreamAllUserChanges) propagate space_id through to clients.
- changeFromRow surfaces SpaceID on the wire Change shape.
- New extractSpaceID helper reads the incoming payload — prefers top-
  level spaceId, falls back to data.spaceId (inserts) or
  fields.spaceId.value (updates). Tolerates pre-v28 clients.
- 6 Go tests cover the helper + round-trip.

Client-side:
- PendingChange gains an optional spaceId.
- Dexie creating hook stamps spaceId from the active record onto the
  pending-change row (already set by the v28 scope hook).
- Dexie updating hook reads spaceId from the pre-update record and
  stamps it on the pending-change so updates carry space context even
  though spaceId itself is immutable and never in `fields`.
- buildChangeset forwards spaceId to the server.

Explicitly NOT in scope this pass:
- RLS remains user_id-scoped; multi-member shared-space reads need a
  second policy that joins against auth.members. Follow-up once shared
  spaces are actually used — today everything is personal.
- Subscription fan-out is still per-user; fan-out to all members of a
  shared space is part of the same follow-up.

Go tests: 6/6 pass. Web type-check clean (0 errors across 7139 files).

Plan: docs/plans/spaces-foundation.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 16:53:14 +02:00
Till JS
1cd559ca34 feat(mana-ai): server runner on runPlannerLoop, drops text-JSON parser
Migrates the background tick from buildPlannerPrompt + PlannerClient +
parsePlannerResponse to the shared runPlannerLoop with native function
calling. Structurally identical to the webapp runner (commit 5a) —
same catalog, same compact system prompt, same multi-turn chat.

Server-specific twist: the ``onToolCall`` callback is a no-op stub
(returns {success:true, message:'recorded — pending client
application'}). The server has no Dexie access, so it can't actually
execute writes; instead it captures the LLM's chosen tool_calls and
writes them as PlanStep entries on the iteration. The user's client
picks up those planned steps on sync — same shape as before, just
sourced from the LLM's native tool_calls instead of a regex-extracted
JSON block.

Scope trimmed by the SERVER_TOOLS filter: only propose-default (write)
tools go to the server planner. Read-only tools (list_*, get_*) are
hidden because stubbing a response would let the LLM hallucinate that
it saw real data. Read-then-act chains stay with the foreground
runner, which has a real executor.

Deleted: planner/client.ts (old PlannerClient; replaced by
planner/llm-client.ts). Drift guard in tools.ts collapses into a
SERVER_TOOLS = AI_TOOL_CATALOG.filter(propose) derivation — no more
hand-maintained duplicate list; the contract test now asserts the
inverse round-trip against AI_PROPOSABLE_TOOL_SET.

TODO (follow-up): token usage tracking is temporarily set to 0 because
runPlannerLoop doesn't expose per-message usage yet. Budget
enforcement on the server is effectively disabled until the loop
returns that data — the webapp runner is unaffected.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 16:39:20 +02:00
Till JS
698ffe797c feat(spaces): add spaces pg schema — credentials + module_permissions
Groundwork for server-side Space extensions that must NOT live in Dexie:
  - spaces.credentials         — per-space OAuth tokens, API keys, SMTP
                                 configs. Access tokens are stored
                                 encrypted at rest with the service KEK.
  - spaces.module_permissions  — role × module read/write/admin overrides
                                 on top of the SPACE_MODULE_ALLOWLIST
                                 defaults.

Both tables FK to auth.organizations with ON DELETE CASCADE so deleting
a space drops its credentials and permission overrides automatically.

RLS is intentionally deferred — enabling it now would lock out services
that don't yet pass space context. A follow-up migration turns it on
after mana-api speaks the Spaces protocol end-to-end.

To apply locally: bun run db:push in services/mana-auth, or psql -f
sql/004_spaces.sql against the mana_platform DB.

No runtime code reads these tables yet — they're the scaffolding that
Task-8 (mana-sync) and the eventual social-relay/clubs modules will
consume.

Plan: docs/plans/spaces-foundation.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 16:13:33 +02:00
Till JS
da1bb2d6e9 feat(spaces): auto-create personal space on signup via Better Auth hook
Wires databaseHooks.user.create.after to call createPersonalSpaceFor,
which provisions a Better Auth organization of type='personal' with the
user as owner. Every signup now produces a usable default space — no
UI code needed to bootstrap it.

Details:
- Slug derived from email local-part, lowercase, alphanumerics + hyphens,
  max 30 chars, random fallback if nothing usable remains.
- Reserved-slug list (me/admin/api/auth/…) blocks system-route clashes.
- Collision resolver appends -2, -3, … up to 999 before falling back to
  a random suffix. Tests cover both the DB-taken and reserved-slug cases
  via an injectable SlugTakenLookup (no DB needed for unit tests).
- Idempotent: if a personal space already exists for the user, returns
  it instead of creating a duplicate. Guards against retry double-signup.
- Failure propagates — an orphan user without a personal space is worse
  than a retry-able signup error.

Existing dev users will need a backfill or a re-provisioning of the dev
DB — new users are unaffected.

12 tests pass (23 total across the spaces module).

Plan: docs/plans/spaces-foundation.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 16:10:26 +02:00
Till JS
166d6c6ffb feat(spaces): validate space metadata on Better Auth organization hooks
Moves the canonical SpaceType + SPACE_MODULE_ALLOWLIST to @mana/shared-types
(framework-free) so the Bun services can consume them without pulling in
Svelte. shared-branding keeps only the UI-facing labels and descriptions
and re-exports the canonical types for frontend convenience.

Wires two Better Auth organization hooks in mana-auth:
- beforeCreateOrganization asserts metadata.type is a valid SpaceType,
  rejecting the create with a BAD_REQUEST otherwise.
- beforeDeleteOrganization rejects deletion of the personal space.

Covered by bun tests (11 assertions) for the helper module.

No migration and no schema change — type lives in the existing
organization.metadata jsonb column.

Plan: docs/plans/spaces-foundation.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 16:05:38 +02:00
Till JS
e757470cb0 feat(mana-llm): add OpenAI-style tools + tool_calls passthrough
Extends the chat-completions surface so callers can ask any provider
to call named functions and get structured tool_calls back. Wired
through all three provider adapters so the planner and companion can
switch off the fragile JSON-parsing pathway.

- Request: tools[], tool_choice, assistant tool_calls, tool-role
  messages with tool_call_id.
- Response: MessageResponse.tool_calls, Choice.finish_reason adds
  "tool_calls", DeltaContent streams tool_calls.
- Google provider: Tool(function_declarations=...) build, result
  normalised (args dict → JSON string), function_response parts on
  a user turn for tool-role messages.
- OpenAI-compat: 1:1 passthrough of the OpenAI spec.
- Ollama: /api/chat passthrough; model-level capability check via a
  TOOL_CAPABLE_OLLAMA_PATTERNS whitelist (llama3.1+, qwen2.5+,
  mistral, command-r, …) — unsupported models rejected rather than
  silently falling back to prose.
- Router: model_supports_tools() check upfront for both streaming
  and non-streaming paths; ProviderCapabilityError bubbles as 400.

No silent downgrade. Missing tool support = explicit error.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 15:22:48 +02:00
Till JS
c612a22371 fix(type-check): unblock two more pre-existing failures
After yesterday's type-check cascade repair (c34175afa), the root
\`pnpm run type-check\` progressed through 5 more packages but still
stopped on two pre-existing failures:

- \`services/mana-media\` delivery route: \`c.body(transformedBuffer)\`
  passed a Node \`Buffer<ArrayBufferLike>\`, but Hono 4.7 types the
  body argument as \`Uint8Array<ArrayBuffer>\` (strict — no
  ArrayBufferLike). \`Uint8Array.from(buf)\` gives a clean copy with a
  fresh \`ArrayBuffer\` backing that the strict type accepts. Runtime
  cost for a handful of KB per image transform is negligible next to
  the Sharp pipeline that produced the buffer.
- \`packages/shared-llm\`: same rune issue as local-stt + local-llm —
  \`store.svelte.ts\` uses \`$state\` and transitively pulls in
  \`local-llm/src/svelte.svelte.ts\`. Plain tsc can't resolve Svelte 5
  runes. Same treatment: \`type-check\` script explicitly skips with a
  message pointing at svelte-check.

Root \`pnpm run type-check\` now reaches \`@context/mobile\`, which has
real code-level type errors (adapter shape mismatches, an RN event-
handler typing drift, and a deleted Supabase module still imported by
\`utils/supabaseTest.ts\`). Those need domain changes, not config
tweaks — out of scope for this repair pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 15:20:08 +02:00
Till JS
4b8fede7fc fix(mana-llm): surface Gemini finish_reason errors instead of returning ""
The google provider called response.text after a chat completion and
passed the resulting string downstream unchanged. When Gemini's content
filter, recitation guard, or max_tokens ceiling fired, response.text
quietly returned "" — which the planner then reported as "no JSON block
found", masking the real cause. Empirically this failed in 45 ms on a
simple Quiz mission.

Introduces providers/errors.py with a small ProviderError hierarchy
(Blocked / Truncated / Auth / RateLimit / Capability). google.py now
inspects response.candidates[0].finish_reason and raises the matching
structured error; the non-streaming path maps it to 422/502/429 via a
new except-branch in main.py, and the streaming path surfaces the kind
as the SSE error type. Capability is wired but not yet used — it lands
with the tool-schema passthrough in the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 15:15:37 +02:00
Till JS
5ec1dfc747 chore(db): enforce pgSchema isolation with a lint script
The "every Drizzle table uses pgSchema" rule was documented in
.claude/guidelines/database.md (added yesterday as part of Concern 5)
but enforced only by convention. A new service could slip a raw
\`pgTable()\` past review and collide in the default \`public\` schema
of \`mana_platform\`, and nothing would surface the mistake until a
production migration failed.

- \`scripts/validate-pg-schema-isolation.mjs\` scans every tracked
  TypeScript file under services/, apps/api/, packages/ for call sites
  of \`pgTable(\` (not imports — imports can still be useful for types).
  Strips comments before matching so doc-examples like "use \`pgTable()\`"
  don't trigger false positives.
- Wired as \`pnpm run validate:pg-schema\` and a new CI step in the
  validate job (right after the turbo-recursion check). 721 files
  scan clean today.
- Removed an unused \`pgTable\` import in mana-subscriptions that would
  have been the only import of the symbol remaining after this change.
- Updated .claude/guidelines/database.md — the old verification blurb
  said "no automated lint rule yet", now points at the enforcer.

Drift verified: injecting a synthetic \`pgTable('bad', {})\` into
subscriptions.ts failed with a clear file:line violation pointing at
the database guideline.

Closes the "no automated lint rule" gap noted in the database guideline.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 14:45:59 +02:00
Till JS
52d008dd34 fix(goals): start GoalTracker on boot + surface AI proposals inline
startGoalTracker was only ever called from tests, so DrinkLogged /
TaskCompleted / MealLogged events never incremented currentValue and
GoalReached never fired — the progress bars were cosmetic. Wire it into
the (app)/+layout idle boot next to startStreakTracker, with matching
teardown in onDestroy.

Also drop <AiProposalInbox module="goals"/> into the module ListView so
create_goal / pause_goal / resume_goal / complete_goal proposals are
reviewable inline (previously only visible in the mission-detail view).

Refresh the tool-coverage tables while we're at it: apps/mana/CLAUDE.md
now reflects the real catalog state (59 tools, 19 modules — was 37/12),
and services/mana-ai/CLAUDE.md shows the correct server-side propose
subset (31 tools, 16 modules). Also fixes a stale 'location_log' →
'get_current_location' typo in the places row.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-20 14:24:39 +02:00
Till JS
a7fe828d32 refactor(auth): extract sso-origins SSOT + harden drift test
TRUSTED_ORIGINS was defined inside better-auth.config.ts, which pulls
in the whole Better Auth stack just to read a list of hostnames. Anyone
who wants to consume the list (infra tooling, compose-env generators,
monitoring) had to either duplicate it or pay the import cost.

- New `sso-origins.ts` — zero-dep module exposing
  `PRODUCTION_TRUSTED_ORIGINS` + `LOCAL_TRUSTED_ORIGINS` + the combined
  `TRUSTED_ORIGINS` list. This is now the canonical place to add a new
  top-level SSO origin.
- `better-auth.config.ts` imports + re-exports so existing consumers
  keep working without a touch.
- `sso-config.spec.ts` imports directly from `./sso-origins` (cleaner
  coupling) and now HARD-FAILS when mana-auth CORS_ORIGINS contains a
  production origin that isn't in trustedOrigins. Previously this was
  a `console.warn` only, meaning dead-drift could silently accumulate
  and then surface as a confusing runtime auth rejection.
- Root CLAUDE.md "Adding an app to SSO" updated to point at the SSOT
  and mention the new hard-fail direction.

No current drift — the mana-auth CORS_ORIGINS already match. The
hardened assertion is defensive for future changes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 19:45:42 +02:00
Till JS
8dd3dbc9e5 docs(mana-research): step-by-step API_KEYS.md setup guide
Complete walkthrough per provider — signup URL, free-tier details,
pay-per-use pricing, env-var name, key format — plus sections on where
to paste keys (.env.secrets), BYO-keys vs server-keys, verification
curl commands and troubleshooting (including the cross-service
MANA_SERVICE_KEY mismatch encountered during live testing).

Linked from services/mana-research/CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 18:50:06 +02:00
Till JS
85537cb92a fix(research): default Gemini to 2.5-flash (2.0-flash deprecated for new users)
Google deprecated `gemini-2.0-flash` for new API users — existing
accounts still work, but a freshly-billed key returns 404
"models/gemini-2.0-flash is no longer available to new users". The
working replacement is `gemini-2.5-flash` (same price tier, better
quality, groundingMetadata shape unchanged).

Verified live: the fix produced a real answer with 6 grounding
citations in 2.6s.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 17:01:24 +02:00
Till JS
97abd251e3 fix(events): Eventbrite provider — switch from dead API to web scraping
Eventbrite shut down their public Event Search API (/v3/events/search)
in 2023. The provider now uses the website extractor pipeline
(mana-research + LLM) to scrape Eventbrite's public search pages.
No API key needed — same pipeline as any website source.

Also adds mana-events to generate-env.mjs for automatic .env generation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 16:51:58 +02:00
Till JS
536fc89050 fix(research): Claude Opus 4.7 rejects temperature param + log executor errors
- claude-web-search.ts: only send `temperature` when caller explicitly
  sets one. Opus 4.7 deprecated the param and returns
  400 invalid_request_error "`temperature` is deprecated for this
  model." Sonnet/Haiku still accept it, so keep the opt-in path.
- execute-research.ts: log provider errors via console.warn so future
  integration failures are visible in stdout. Previously the executor
  swallowed the underlying error and only returned a generic errorCode,
  which made diagnosing vendor-specific API changes impossible.

Discovered via smoke-testing with a real Anthropic key — the direct
curl worked, but our provider 400'd because Opus 4.7 tightened the
accepted param set.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-18 16:36:22 +02:00
Till JS
ed801cf725 feat(events): Phase 4 — provider adapters for Eventbrite + Meetup
- Add EventProvider interface (base.ts) with fetchEvents(url, name, ctx, config)
- Refactor iCal parser and website extractor as provider adapters
- Add Eventbrite provider: API v3 search by location, category mapping,
  price info extraction. Requires EVENTBRITE_API_KEY env var.
- Add Meetup provider: GraphQL API search by location, topic→category
  mapping, HTML stripping. Requires MEETUP_API_KEY env var.
- Provider registry (getProvider, PROVIDER_TYPES) replaces hardcoded
  switch in crawl-scheduler
- Crawl scheduler now joins sources with regions for ProviderContext
  (lat/lon/radius/label) — platform providers need this for geo-search
- Source creation accepts 'eventbrite' and 'meetup' types (url optional)
- Both providers gracefully return empty when API keys unconfigured

116 tests (all passing), no regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 15:43:40 +02:00
Till JS
2c0d866287 feat(events): Phase 3 — AI tools, Event-Scout template, feedback loop
- Add discover_events (auto) and suggest_event (propose) to shared-ai
  tool catalog. discover_events reads the discovery feed, suggest_event
  creates a proposal to save a discovered event to the user's calendar.
- Add Event-Scout agent template with daily "Events der Woche" mission.
  Policy: discover_events=auto, suggest_event=propose, all else denied.
- Add frontend tool implementations in events/tools.ts — discover_events
  calls the feed API, suggest_event delegates to discoveryStore.saveEvent.
- Add feedback.ts — computes implicit user profile from save/dismiss
  history (category affinity + source quality as 0–2x weight multipliers).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 15:37:28 +02:00
Till JS
b5d55fdb21 feat(events): add Event Discovery — Phase 1 + 2
Phase 1: Manual iCal feeds + Discovery tab
- 5 new DB tables in event_discovery schema (regions, interests,
  sources, discovered_events, user_actions)
- iCal parser (node-ical) with deduplication (SHA-256 hash)
- Crawl scheduler (15-min interval, auto-deactivate after 5 errors)
- CRUD routes for regions, interests, sources + paginated feed endpoint
- Frontend: "Meine Events" / "Entdecken" tab navigation in ListView
- Discovery setup wizard (regions via mana-geocoding + interests)
- DiscoveredEventCard with save/dismiss, SourceManager for iCal feeds
- "Merken" creates a local socialEvent from discovered event

Phase 2: Auto source discovery + LLM extraction + relevance scoring
- Source discoverer: web search via mana-research to auto-find iCal
  feeds and venue websites for a region
- Website extractor: crawl via mana-research /extract, then LLM-based
  event extraction via mana-llm with structured JSON output
- Flexible date parsing (ISO, DD.MM.YYYY), markdown fence stripping
- Relevance scorer: category match, freetext match, haversine distance,
  time proximity, weekend bonus (0-100 clamped)
- Routes: POST regions/:id/discover-sources, PUT/DELETE sources/:id/activate|reject
- Frontend: "Automatisch finden" button, suggested vs active sources UI

107 tests (all passing), no regressions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 15:30:46 +02:00
Till JS
7d120225dc feat(research): Phase 3b openai-deep-research async + BYO-keys CRUD & UI
Two backlog items landed in one commit because an earlier amend in a
parallel terminal dropped the initial Phase 3b commit and the BYO-keys
work was blocked on the same wiring.

openai-deep-research (async):
- New research.async_jobs table persists the OpenAI response.id, query,
  reservation, and cached result/error.
- POST /v1/research/async reserves credits, submits to the Responses API
  with background=true, returns a taskId. Submit failure refunds.
- GET /v1/research/async/:taskId polls upstream, commits the reservation
  on completion, refunds on failure, short-circuits for terminal states.
- GET /v1/research/async lists the user's async tasks.

BYO-keys:
- research.provider_configs CRUD at /v1/provider-configs. Keys are masked
  (••••last4) on read so the raw secret never re-transits to the browser.
  Currently stored plaintext with a TODO for AES-GCM-256 via the shared
  KEK — single call site in storage/configs.ts.decryptKey().
- New frontend route /research-lab/keys lets the user paste a key per
  provider, toggle enabled, and set daily/monthly credit budgets.
- ListView grew a 🔑 link in the header.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 15:43:12 +02:00
Till JS
8f0a74b2e7 feat(research-lab): tier gate (beta+), 1–5 star ratings, run detail route
- Branding: research-lab registered in @mana/shared-branding with requiredTier: 'beta' + a custom flask-on-purple icon, so guest/public users are filtered out of the workbench picker.
- Backend: compare routes now return resultId alongside each CompareEntry so the frontend can wire ratings to the eval_results rows in research.*.
- Frontend: click-to-rate stars in CompareColumn (persists via POST /v1/runs/:runId/results/:resultId/rate), recent-run list rows are now buttons that navigate to /research-lab/runs/[id], and the detail route reconstructs CompareEntry shapes from eval_results + reuses CompareColumn for a full read-only view of any past run.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 15:28:02 +02:00
Till JS
8823cc0bf0 feat(profile): voice interview with pre-rendered TTS audio + Orpheus/Zonos backends
Voice-based interview for the profile module — users choose between text,
voice (question read aloud + mic for answer), or conversation mode (fully
automatic flow with auto-save).

Interview audio:
- 92 pre-rendered MP3 files (23 questions × 4 voices) via Edge TTS
- Voices: Seraphina (DE-f), Florian (DE-m), Leni (CH-f), Jan (CH-m)
- User picks voice via dropdown, persisted in localStorage
- Web Speech API fallback for missing audio files

Profile UI:
- Interview hero block on overview with 3 start modes (text/voice/conversation)
- Voice/conversation toggle + voice picker in interview view
- Mic button on text/textarea/tags inputs for per-question voice input
- Conversation mode: auto-save + auto-advance after STT transcription
- Recording/transcribing/speaking state indicators

mana-tts service:
- New Orpheus TTS backend (German finetune, SNAC codec)
- New Zonos TTS backend (Zyphra, 200k hours, emotion control)
- Endpoints: POST /synthesize/orpheus, POST /synthesize/zonos
- espeak-ng installed on GPU server for Zonos phonemizer
- Compare script for side-by-side voice quality testing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 15:22:52 +02:00
Till JS
49f315f6be feat(research): Phase 3a — 4 sync research agents
Adds Perplexity Sonar, Claude web_search, OpenAI Responses, and Gemini
Grounding as ResearchAgents behind the same comparison interface as the
search and extract providers.

New endpoints:
  POST /v1/research          — single-agent (or auto-routed to the first
                               provider with a configured key)
  POST /v1/research/compare  — fan-out across N agents, persist all
                               answers + citations in research.eval_*

Each agent normalizes its native response into a common AgentAnswer shape
(answer text + citations[] + tokenUsage), storing the provider's raw
response alongside for later inspection. Implementations use direct HTTP
against each vendor's public API — no SDK deps added.

Auto-routing preference: perplexity-sonar → gemini-grounding →
openai-responses → claude-web-search → (openai-deep-research stubbed for
Phase 3b). Credits orchestration reuses the search/extract executor
pattern (reserve → call → commit/refund).

Deferred to Phase 3b: openai-deep-research (async job queue), migration
of mana-ai + mana-api news-research to call this service directly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 15:06:12 +02:00
Till JS
2bdb48bdd1 feat(research): add mana-research service — Phase 1 + 2
New Bun/Hono service on port 3068 that bundles many web-research providers
behind a unified interface for side-by-side comparison. All eval runs
persist in research.* (mana_platform) so quality can be reviewed later.

Providers (Phase 1+2):
  search:  searxng, duckduckgo, brave, tavily, exa, serper
  extract: readability (via mana-search), jina-reader, firecrawl

Endpoints:
  POST /v1/search, /v1/search/compare       — single + fan-out
  POST /v1/extract, /v1/extract/compare     — single + fan-out
  GET  /v1/runs, /v1/runs/:id               — history
  POST /v1/runs/:run/results/:id/rate       — manual eval
  GET  /v1/providers, /v1/providers/health  — catalog + readiness

Auto-routing: when `provider` is omitted, queries are classified via regex
(fast path, 0ms) with optional mana-llm fallback, then routed to the first
available provider for that query type (news → tavily, academic → exa,
semantic → exa, etc.).

Credits: server-key calls go through mana-credits reserve → commit/refund
so failed provider calls don't charge the user. BYO-keys supported via
research.provider_configs (UI arrives in Phase 4).

Cache: Redis with graceful degradation (1h TTL for search, 24h for
extract). Pay-per-use APIs only — no subscription-gated providers.

Docs: docs/plans/mana-research-service.md + docs/reports/web-research-capabilities.md

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 14:42:25 +02:00
Till JS
004fc0b2fd feat(credits): add 2-phase debit (reserve/commit/refund)
Introduces credit_reservations table + three internal endpoints so services
that need to charge only after a downstream call succeeds (notably the
upcoming mana-research fan-out across paid provider APIs) can reserve
credits atomically, then commit on success or refund on failure. One-shot
/credits/use remains for synchronous operations.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-17 14:41:41 +02:00
Till JS
76577869e1 feat(mana-ai): OpenTelemetry tracing + Grafana Tempo backend
Add distributed tracing to the mana-ai background runner so mission
execution can be visualized end-to-end in Grafana.

Instrumentation (services/mana-ai/):
- tracing.ts: OTel provider setup with OTLP/HTTP exporter, withSpan() helper
- tick.ts: tick.planMission span with mission/agent/user attributes
- client.ts: planner.complete span with LLM model, tokens, latency

Infrastructure:
- docker/tempo/tempo.yaml: Grafana Tempo config (OTLP HTTP on 4318)
- docker-compose: tempo service + tempo_data volume + mana-ai env var
- docker/grafana/provisioning/datasources/tempo.yml: auto-provisioned

Trace flow:
  tick.planMission (root span)
    └── planner.complete (child span)
        ├── llm.model = "gpt-4o-mini"
        ├── llm.tokens.total = 1234
        └── llm.response.length = 567

Enable: set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
View: Grafana → Explore → Tempo datasource

Also fixes: removed broken @mana/subscriptions workspace ref from arcade.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 15:21:23 +02:00
Till JS
acd7e0d6b0 docs: update architecture comparison — 5/10 roadmap items done
Update report to reflect all completed work:
- Matrix: streaming , tool registration updated to 29 tools + MCP
- §5.2 Streaming: marked done
- §5.3 Tool System: marked done
- §6 Table: items 1-3 + 5 struck through with commit refs
- §8 Fazit: updated gaps and recommendations

5 of 10 roadmap items complete in one session:
1. SSE Streaming, 2. Dynamic Tool Registry, 3. Budget Enforcement,
5. MCP Server Export (27/29 tools with DB ops), plus Tool Drift Fix.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 15:00:09 +02:00
Till JS
ce57e11950 feat(mana-ai): server-side token budget enforcement per agent
Implement rolling 24h token budget enforcement in the mana-ai tick loop.
Agents with maxTokensPerDay set are now rate-limited server-side.

Changes:
- PlannerClient: extract usage.total_tokens from mana-llm response
- planOneMission: return {plan, tokensUsed} tuple
- tick loop: check getAgentTokenUsage24h() before planning; skip with
  'skipped-budget' decision if over limit
- tick loop: record token usage after successful plan via
  recordTokenUsage() INSERT into mana_ai.token_usage
- migrate.ts: new mana_ai.token_usage table with rolling window index
- metrics.ts: mana_ai_tokens_used_total counter (by agent_id)

Budget flow:
  Agent.maxTokensPerDay = 50000
  → tick checks: SELECT SUM(tokens_used) WHERE ts > now()-24h
  → if sum >= 50000: skip mission, emit skipped-budget metric
  → else: plan mission, INSERT token_usage row

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 14:41:31 +02:00
Till JS
d40a61119e refactor(ai): dynamic tool registry — single-source catalog in shared-ai
Introduce AI_TOOL_CATALOG in @mana/shared-ai as the single source of truth
for all 29 tool schemas (17 propose + 12 auto). Both the webapp policy and
the server-side mana-ai planner now derive their tool lists from the catalog
instead of maintaining independent hardcoded copies.

- New: packages/shared-ai/src/tools/schemas.ts — catalog with ToolSchema type
- Rewrite: proposable-tools.ts — derived from catalog instead of hardcoded array
- Rewrite: services/mana-ai/src/planner/tools.ts — 277→30 lines (imports from catalog)
- Simplify: webapp policy.ts — derives AUTO/PROPOSE from catalog defaultPolicy

Adding a new tool now requires 2 files instead of 3-5:
1. Add schema to AI_TOOL_CATALOG (shared-ai)
2. Add execute function in the module's tools.ts (webapp)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 13:06:07 +02:00
Till JS
3be4612f04 fix(mana-llm): google-genai v1.73 keyword-only Part.from_text()
google-genai >=1.70 changed Part.from_text() from positional to
keyword-only argument. The production container installed v1.73.1
and crashed on startup with "Part.from_text() takes 1 positional
argument but 2 were given".

Fix: Part.from_text(msg.content) → Part.from_text(text=msg.content)

Tested live: curl https://llm.mana.how/v1/chat/completions with
model=google/gemini-2.5-flash returns correct response.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 12:47:23 +02:00
Till JS
659a7d9774 fix(mana-llm): add google-genai to requirements.txt for Docker builds
google-genai was in pyproject.toml but missing from requirements.txt.
The Dockerfile uses pip install -r requirements.txt, so the Google
provider never loaded in production. Now that the key is set and the
cloud tier upgraded to gemini-2.5-flash, the import fires on startup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 12:40:30 +02:00
Till JS
d83fc370a0 docs: update tool coverage table + server-side research + templates
Catches up all docs with the current state of the AI tool system.

services/mana-ai/CLAUDE.md:
- New v0.6 status section documenting NewsResearchClient,
  pre-planning research injection, config.manaApiUrl, and the full
  28-tool / 11-module inventory (17 propose + 11 auto).

apps/mana/CLAUDE.md:
- New "Tool Coverage" table in the AI Workbench section listing all
  tools per module with their policy (propose vs auto).
- New "Templates" subsection documenting the two-section gallery
  (agent vs workbench templates), the seed-handler registry, and
  the current handlers (meditate, habits, goals).
- Architecture cross-reference updated to include §23.

docs/architecture/COMPANION_BRAIN_ARCHITECTURE.md:
- §23.2 gains a "Server-Side Research (mana-ai, ab v0.6)" subsection
  explaining how NewsResearchClient mirrors the client-side research
  pre-step: same endpoints, same trigger regex, but HTTP-direct from
  the Docker network instead of SvelteKit-internal.

docs/plans/README.md:
- workbench-templates.md added to the roadmap table (T1 shipped).
- Multi-agent description updated to mention 28 tools + server-side
  web-research.
- Architecture cross-reference includes §23.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 12:35:40 +02:00
Till JS
8a0bf93699 chore(cloud-tier): upgrade default model gemini-2.0-flash → gemini-2.5-flash
gemini-2.0-flash is deprecated June 1 2026. gemini-2.5-flash has been
stable since Q1 2026 with similar pricing ($0.15/$0.60 per 1M tokens
vs $0.10/$0.40 — pricing table already had the entry).

Three files touched:
- packages/shared-llm/src/backends/cloud.ts — client default
- services/mana-llm/src/config.py — server default
- services/mana-llm/src/providers/google.py — Ollama→Gemini fallback
  map + constructor default + deduplicated model list

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 12:32:03 +02:00
Till JS
23b8cc13fb feat(ai-tools): server-side web-research + contacts for agents
Two major tool expansions — the Recherche-Agent and Today-Agent can
now research the web autonomously (no browser needed), and a future
Meeting-Prep agent can read + create contacts.

=== research_news (server-side execution) ===

The biggest addition: mana-ai can now call mana-api's news-research
endpoints (POST /discover + /search) directly, without a browser.

Infrastructure:
- services/mana-ai/src/planner/news-research-client.ts — full HTTP
  client with discover→search pipeline. 15s/30s timeouts. Graceful
  null on any failure (network, mana-api down, bad response) so the
  tick never crashes from research errors.
- config.manaApiUrl added (default http://localhost:3060); wired in
  docker-compose.macmini.yml as http://mana-api:3060 + depends_on
  mana-api with service_healthy condition.

Pre-planning research step (cron/tick.ts):
- Before the planner prompt is built, the tick checks if the
  mission's objective or conceptMarkdown matches research keywords
  (same RESEARCH_TRIGGER regex the webapp uses). When it matches:
  * NewsResearchClient.research(objective) runs discovery + search
  * Results are injected as a synthetic ResolvedInput with id
    '__web-research__' and a formatted markdown context block
  * The Planner then sees real article URLs/titles/excerpts and can
    reference them in create_note / save_news_article steps
  * Log line: "pre-research: N feeds, M articles"

Tool registration:
- research_news added to AI_PROPOSABLE_TOOL_NAMES + mana-ai tools.ts
  with params (query, language?, limit?). This lets the planner also
  explicitly propose a research step as a PlanStep (in addition to
  the pre-planning auto-injection).

=== create_contact ===

- Added to AI_PROPOSABLE_TOOL_NAMES + mana-ai tools.ts with params
  (firstName required, lastName/email/phone/company/notes optional).
- Contacts are encrypted at rest; server planner can plan the step
  but execution stays on the webapp (same as all propose tools).
  Full server-side contact resolution via Key-Grant is a future
  enhancement.
- get_contacts added to webapp AUTO_TOOLS so agents can inspect
  existing contacts without nagging (read-only, auto-policy).

Module coverage now:
   todo (5)   calendar (2)   notes (5)   places (4)
   drink (3)  food (2)       news (1)    journal (1)
   habits (3)  news-research (1)  contacts (1)

  11 modules, 28 tools total (17 propose, 11 auto).

Tests: mana-ai 41/41 (drift-guard passes), shared-ai type-check
clean, webapp svelte-check 0 errors, 0 warnings.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 12:25:45 +02:00
Till JS
1266b583e4 feat(ai-tools): unlock create_note + create_journal_entry + habit tools for agents
Closes the three biggest tool-coverage gaps so the shipped agent
templates can actually do their job end-to-end. Before this, the
Recherche-Agent couldn't create notes (only edit), the Today-Agent
couldn't create journal entries, and no habit-related tool was
server-proposable at all.

shared-ai (proposable-tools.ts):
- create_note (notes) — key unlock: Recherche-Agent now creates
  per-source notes and the summary report.
- create_journal_entry (journal) — key unlock: Today-Agent proposes
  a poem as a journal entry with optional mood.
- create_habit (habits) — agent can suggest new habits.
- log_habit (habits) — agent can log a habit completion for today.

Organized the list with per-module section comments for readability
now that we're at 15 proposable tools.

mana-ai (planner/tools.ts):
- 5 new tool definitions with full parameter schemas:
  * create_note (title, content?)
  * create_journal_entry (content, title?, mood? enum)
  * create_habit (title, icon, color)
  * log_habit (habitId, note?)
- Drift-guard contract test passes (41/41) — confirms the mana-ai
  tool list is in sync with the shared-ai canonical set.

Webapp (policy.ts):
- get_habits added to AUTO_TOOLS (read-only; agent can inspect
  which habits exist without nagging the user for approval).
- list_notes added to AUTO_TOOLS (was already used in the reasoning
  loop but missing from the explicit auto-list; the planner default
  fell through to 'propose' which was wasteful for a read op).

Module coverage after this change:
   todo (5 tools)   calendar (2)   notes (5 incl. create)
   places (4)       drink (3)      food (2)
   news (1)         journal (1)    habits (3)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 12:00:17 +02:00
Till JS
4d9b16a683 feat(notes): list + update + append + add_tag tools for the AI
Makes the "read all notes and tag them #Natur/#Technologie/…" use case
fully functional. Four new ModuleTool entries in notes/tools.ts:

- list_notes(limit?, query?, includeArchived?) — auto, read-only. Returns
  id + title + excerpt so the planner can reference concrete notes
  without dumping full bodies.
- update_note(noteId, title?, content?) — proposable. Destructive full
  overwrite. Docstring nudges toward append_to_note when applicable.
- append_to_note(noteId, content) — proposable, non-destructive. Handles
  the trailing-newline separator so markdown stays clean.
- add_tag_to_note(noteId, tag) — proposable, idempotent, case-insensitive.
  Strips leading #, replaces spaces with _, skips if already present.
  Exactly the categorization primitive the user asked for.

All three writes are added to AI_PROPOSABLE_TOOL_NAMES so both the
webapp policy and mana-ai's boot-time drift guard agree (now 11 tools).
Mirrored in services/mana-ai/src/planner/tools.ts.

AiProposalInbox mounted on /notes so approvals land inline in the
notes module too (already appears in the mission-detail cross-module
inbox via the earlier commit).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-16 00:24:48 +02:00
Till JS
b768a0ffce refactor(shared-rss): extract RSS parsing + Readability into one package
news-ingester and apps/api both shipped their own copy of rss-parser
+ jsdom + Readability glue. Single source now in packages/shared-rss.
Adds discoverFeeds (rel=alternate + common-paths probe) and validateFeed
which News Research will use. JSDOM virtualConsole is silenced once,
in the package, instead of in two parallel call sites.

- packages/shared-rss: parse, extract, discover, validate
- services/news-ingester: drop local parsers, depend on @mana/shared-rss
- apps/api: drop @mozilla/readability + jsdom direct deps, use shared

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 22:30:44 +02:00
Till JS
e8077a44b4 docs: flesh out Multi-Agent plan shipping log + Team Workbench forward plan
The Multi-Agent Workbench shipped end-to-end (commits 1771063df through
7c89eb625). This commit turns the plan doc into a proper history + post-
mortem and captures the deferred Team-Workbench as its own forward plan
so the architectural breadcrumbs don't rot.

docs/plans/multi-agent-workbench.md:
- Status bumped to  Shipped; every phase checkbox flipped.
- Open-questions section rewritten with the decisions that were
  actually made (name-unique via store write-time check, per-source
  system principalIds, policy fully migrated, scene binding default-
  empty with smart suggestion).
- New "Shipping-Historie" table mapping each phase to its commit, the
  number of files touched, and the test outcome.
- New "Lessons Learnt + Follow-Up Ideen" with:
  * What went better than expected (L3 Actor cutover, getOrCreate
    instead of unique index, displayName caching)
  * Thin spots worth revisiting (avatar not on Actor, missing token
    counter for budget, no missions list on agent detail, no
    drag-reassign, scene binding doesn't drive filters yet)
  * Five deferred follow-up projects (team features, agent memory
    self-update, agent-to-agent messaging, meta-planner, per-agent
    encryption domains)

docs/plans/team-workbench.md (NEW):
- Full forward-looking plan for the deferred Team-Workbench.
- Two use-cases (human multi-user vs multi-agent sharing team
  context) with the observation that they share the same infra.
- Decision candidates table (still open — meant as T0 RFC fodder,
  not baked in).
- Architecture sketch with data-model deltas over the current
  single-user shape.
- Encryption subsection dedicated to the hardest problems: team-key
  wrapping per member (reuses Mission-Grant pattern), member-removal
  rotation (lazy vs eager), Zero-Knowledge-mode incompatibility.
- T0..T6 phasing (~7 weeks for a clean first-pass).
- Section "Wie Multi-Agent dafür den Weg geebnet hat" enumerating
  the four invariants the shipped Phase 0-7 deliberately preserved
  to make this plan cheap when it lands.

docs/plans/README.md (NEW):
- Index doc with the AI/Workbench roadmap as an ASCII flow so future
  contributors can locate themselves in the sequence without reading
  three 400-line plans first.

docs/future/AI_AGENTS_IDEAS.md:
- Header marks Point 1 (encrypted tables) as shipped via the Mission
  Grant plan; points 2-8 stay relevant. Cross-link to all three plan
  docs so this stays the go-to backlog.

services/mana-ai/CLAUDE.md:
- Design-context header expanded to link to all four related docs
  (arch §20-22, both shipped plans, forward team plan, ideas backlog).

No code changes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-15 22:17:04 +02:00