managarten/services/mana-sync/CLAUDE.md

mana-sync

Central sync server for local-first Mana apps. Handles data synchronization between IndexedDB (Dexie.js) clients and PostgreSQL via field-level Last-Write-Wins (LWW) conflict resolution.

Architecture

Client A (Browser)                Client B (Browser)
  IndexedDB (Dexie)                 IndexedDB (Dexie)
       |                                 |
       | POST /sync/{appId}              | GET /sync/{appId}/pull
       v                                 v
  ┌──────────────────────────────────────────┐
  │            mana-sync (Go)                │
  │  Port 3050 | JWT auth via JWKS           │
  │                                          │
  │  HTTP: sync + pull endpoints             │
  │  WS:   real-time sync-available notify   │
  │                                          │
  │  Conflict Resolution: Field-level LWW    │
  └──────────────────┬───────────────────────┘
                     |
                     v
               PostgreSQL
            (sync_changes table)

Quick Start

# From monorepo root
pnpm dev:sync           # Start server (requires DB + auth running)
pnpm dev:sync:build     # Compile Go binary

# Standalone
cd services/mana-sync
go build -o server ./cmd/server
JWKS_URL=http://localhost:3001/api/auth/jwks \
DATABASE_URL=postgresql://mana:devpassword@localhost:5432/mana_sync \
./server

Sync Protocol

Push (POST /sync/{appId})

Client sends a batch of changes, server records them and returns changes from other clients.

CLIENT -> SERVER:
{
  "clientId": "chrome-tab-abc123",
  "since": "2024-01-01T10:00:00.000Z",
  "changes": [
    {
      "table": "todos",
      "id": "todo-123",
      "op": "update",
      "fields": {
        "title": { "value": "Buy milk", "at": "2024-01-01T10:05:00Z" },
        "completed": { "value": true, "at": "2024-01-01T10:06:00Z" }
      },
      "actor": { "kind": "user", "principalId": "user-1", "displayName": "Du" },
      "origin": "user"
    }
  ]
}

SERVER -> CLIENT:
{
  "serverChanges": [ ... changes from other clients ... ],
  "conflicts": [],
  "syncedUntil": "2024-01-01T10:06:15.123456789Z"
}

Pull (GET /sync/{appId}/pull)

Client requests changes for a specific collection since a timestamp.

GET /sync/todo/pull?collection=tasks&since=2024-01-01T10:00:00Z
Header: X-Client-Id: chrome-tab-abc123
Header: Authorization: Bearer <jwt>

WebSocket — Unified (GET /ws) [Recommended]

Single connection per user. Receives notifications for all apps with appId in the payload.

CLIENT -> SERVER: { "type": "auth", "token": "<jwt>" }
SERVER -> CLIENT: { "type": "auth-ok" }

// When another client syncs:
SERVER -> CLIENT: { "type": "sync-available", "appId": "todo", "tables": ["tasks"] }

// Keepalive:
CLIENT -> SERVER: { "type": "ping" }
SERVER -> CLIENT: { "type": "pong" }

WebSocket — Legacy (GET /ws/{appId})

One connection per app. Only receives notifications for that specific app. Backward-compatible.

CLIENT -> SERVER: { "type": "auth", "token": "<jwt>" }
SERVER -> CLIENT: { "type": "auth-ok" }
SERVER -> CLIENT: { "type": "sync-available", "appId": "todo", "tables": ["tasks"] }

Conflict Resolution: Field-Level LWW

Each field update carries a timestamp. When the same field is modified by multiple clients, the latest timestamp wins.

Client A: title="Buy milk"    @ 10:05:00
Client B: title="Buy eggs"    @ 10:05:30
Result:   title="Buy eggs"    (Client B wins — later timestamp)

Client A: title="Buy milk"    @ 10:05:00
Client A: completed=true      @ 10:06:00
Client B: title="Buy eggs"    @ 10:05:30
Result:   title="Buy eggs", completed=true  (merged — different fields)

API Endpoints

Endpoint	Method	Auth	Description
POST /sync/{appId}	POST	JWT + Billing	Push changes, get server delta
GET /sync/{appId}/pull	GET	JWT + Billing	Pull changes for a collection
GET /sync/{appId}/stream	GET	JWT + Billing	SSE stream for real-time changes
GET /ws	WS	JWT (in-band)	Unified real-time sync (all apps, one connection)
GET /ws/{appId}	WS	JWT (in-band)	Legacy per-app sync notifications
GET /health	GET	No	Health check with connection stats
GET /metrics	GET	No	Prometheus metrics

Billing gate: Push, pull, and stream endpoints are wrapped by a billing middleware that checks the user's sync subscription status via mana-credits. Returns 402 Payment Required if sync is not active. Status is cached for 5 minutes per user. Fail-open: if mana-credits is unreachable, sync is allowed.

Data Export / Import

Data export is not a mana-sync responsibility anymore (since 2026-04-22). The previous GET /backup/export server-side event-stream export was removed in favour of a fully client-driven snapshot export: the webapp reads its local Dexie store, decrypts per-field, optionally passphrase-seals, and downloads a .mana archive. See apps/mana/apps/web/src/lib/data/backup/v2/ and docs/plans/data-export-v2.md for the format + pipeline.

Rationale for the move:

• Zero-knowledge users hold their vault key client-side only — a server-side exporter cannot produce plaintext archives for them.
• GDPR data-portability is better served by plaintext-by-default (Art. 20) than by ciphertext blobs only decryptable with an active Mana install.
• Module-selective export is intrinsically a client concern — the server has no business knowing which subset of a user's data the user wants to hand out.

Database Schema

Single table for all sync data:

sync_changes (
  id UUID PRIMARY KEY,
  app_id TEXT NOT NULL,
  table_name TEXT NOT NULL,
  record_id TEXT NOT NULL,
  user_id TEXT NOT NULL,
  op TEXT NOT NULL CHECK (insert | update | delete),
  data JSONB,
  field_meta JSONB DEFAULT '{}',
  client_id TEXT NOT NULL,
  created_at TIMESTAMPTZ DEFAULT now(),
  schema_version INT NOT NULL DEFAULT 1,
  actor JSONB,  -- AI Workbench attribution: { kind: user|ai|system, ... }
  origin TEXT,  -- pipeline: user | agent | system | migration
  space_id TEXT
)

actor column (2026-04-14): Opaque JSON blob the webapp stamps on every change to distinguish user writes from autonomous AI writes and derived subsystem writes. Server does NOT parse the shape — just persists + re-emits. Pre-actor clients omit the field; the column is nullable. See apps/mana/apps/web/src/lib/data/events/actor.ts for the discriminated union + COMPANION_BRAIN_ARCHITECTURE.md §20 for the full pipeline.

origin column + field_meta rename (2026-04-26, F1 of docs/plans/sync-field-meta-overhaul.md): field_timestamps was renamed to field_meta for symmetry with the client-side __fieldMeta and to reserve room for richer per-field metadata. The new origin column carries the pipeline that produced the write on the originating client (user / agent / system / migration) — drives client-side conflict-detection: only 'user'-origin writes can lose to a server overwrite and surface a conflict toast (F2). FieldChange wire shape changed from { value, updatedAt } to { value, at } to match.

Indexes: (user_id, app_id, created_at), (table_name, record_id, created_at), (user_id, app_id, table_name, created_at)

Configuration

Variable	Default	Description
PORT	3050	Server port
DATABASE_URL	postgresql://...localhost:5432/mana_sync	PostgreSQL connection
JWKS_URL	http://localhost:3001/api/auth/jwks	mana-auth JWKS endpoint
CORS_ORIGINS	http://localhost:5173,...	Comma-separated allowed origins
MANA_CREDITS_URL	http://localhost:3061	mana-credits service URL for billing checks
MANA_SERVICE_KEY	dev-service-key	Service-to-service auth key

Testing

cd services/mana-sync
go test ./... -v

Test coverage: auth (JWT extraction, validator), config (env loading), sync (validation, serialization, LWW types), backup (ZIP writer round-trip + legacy schema_version=0 clamping + empty-export manifest).

Project Structure

services/mana-sync/
├── cmd/server/main.go          — Entry point, routes, graceful shutdown
├── internal/
│   ├── auth/jwt.go             — EdDSA JWT validation via JWKS
│   ├── auth/jwt_test.go        — Token extraction, validator tests
│   ├── backup/writer.go        — Pure ZIP writer for .mana archives (testable without DB)
│   ├── backup/writer_test.go   — 4 cases: round-trip, empty, legacy schema_version=0
│   ├── backup/handler.go       — HTTP shim for GET /backup/export (auth-only)
│   ├── billing/check.go        — Sync billing status checker (cached, fail-open)
│   ├── config/config.go        — Environment variable loading
│   ├── config/config_test.go   — Config defaults and env override tests
│   ├── store/postgres.go       — PostgreSQL schema, queries
│   ├── sync/handler.go         — HTTP endpoints, LWW logic, validation
│   ├── sync/handler_test.go    — Validation, serialization tests
│   ├── sync/types.go           — Protocol data structures
│   └── ws/hub.go               — WebSocket connection management
├── go.mod
└── CLAUDE.md

Security

• JWT validated via EdDSA JWKS (same as NestJS backends)
• Sync endpoints gated by billing check (402 if subscription inactive)
• WebSocket connections must authenticate within 10 seconds
• Request body limited to 10 MB
• Operation types validated (insert/update/delete only)
• Table and record IDs required on all changes
• RecordChange failures abort the entire sync (no partial writes)
• /backup/export is auth-only by design (GDPR), but StreamAllUserChanges is RLS-scoped to the caller's user_id via the same withUser() transaction pattern as every other query — cross-user export is impossible at the DB layer

Connected Apps (19)

Todo, Calendar, Clock, Contacts, Chat, Questions, Mukke, Context, Photos, Cards, Picture, Presi, Storage, Quotes, SkillTree, CityCorners, Food, Planta, Inventar, uLoad, Times, Calc