mirror of
https://github.com/Memo-2023/mana-monorepo.git
synced 2026-05-14 21:41:09 +02:00
docs(claude): document sync field-meta overhaul (F1-F7) in CLAUDE.md + DATA_LAYER_AUDIT
apps/mana/CLAUDE.md: - Data-flow diagram updated: __fieldMeta + _updatedAtIndex + origin replace the older __fieldTimestamps / __fieldActors / __lastActor trio. - New "Conflict-Detection" sub-section in §Data Layer summarizes the four moving parts (origin-gating, derived updatedAt, server-side singleton bootstrap, stable client_id) with a "use this" cheatsheet for the patterns you'll reach for when writing new module code. DATA_LAYER_AUDIT.md: - Eckdaten line points at v53/v54 instead of "v9 added updatedAt indexes". Conflict-Resolution bullet says "Origin-gated Field-Level LWW via __fieldMeta" (was: __fieldTimestamps). - New "Sync Field-Meta Overhaul (2026-04-26, F1-F7 SHIPPED)" sub-section with one paragraph per phase + commit hash + the four bug-roots that were closed. - Punkt 15 (Conflict-Visualisierung) flipped from "🟢 Backlog" to "✅ Sprint 4+ Backlog C shipped, F2 origin-gated the trigger so only real user edits surface". Future sessions reading the repo cold get the post-overhaul architecture from these two files instead of having to chase the plan + commit log. Closes Punkt 10 of the F1-F7 follow-up audit. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Till JS
parent
899fccd455
commit
98d07a8d48
2 changed files with 58 additions and 15 deletions
|
|
@ -72,10 +72,10 @@ table.add(encryptedRecord) ← Dexie write
|
|||
│
|
||||
▼
|
||||
Dexie hooks (database.ts):
|
||||
- stamp userId
|
||||
- stamp __fieldTimestamps per field
|
||||
- stamp __lastActor + __fieldActors (user / ai / system — see AI Workbench)
|
||||
- record into _pendingChanges (tagged with appId + actor)
|
||||
- stamp userId (user-level tables only)
|
||||
- stamp __fieldMeta[k] = { at, actor, origin } per field
|
||||
- stamp _updatedAtIndex (local-only shadow for indexed sorts)
|
||||
- record into _pendingChanges (tagged with appId + actor + origin)
|
||||
- record into _activity
|
||||
│
|
||||
▼
|
||||
|
|
@ -95,6 +95,28 @@ applyServerChanges → Dexie hooks (suppressed) → liveQuery → decryptRecord
|
|||
|
||||
**Deep dive**: [`apps/web/src/lib/data/DATA_LAYER_AUDIT.md`](apps/web/src/lib/data/DATA_LAYER_AUDIT.md) — sync engine, retry/backoff, quota recovery, telemetry, RLS, encryption rollout, threat model. **Single most important file for understanding how the app works under the hood.**
|
||||
|
||||
### Conflict-Detection (post 2026-04-26 sync-field-meta-overhaul)
|
||||
|
||||
The four bug-roots that made the conflict-toast fire spuriously have all been closed. Architecture today:
|
||||
|
||||
- **`__fieldMeta`** (single hidden field per record, replaces the older `__fieldTimestamps` / `__fieldActors` / `__lastActor` triple). Shape: `{ [field]: { at, actor, origin } }`. The Dexie creating/updating hook stamps it on every write; consumers read it via `readFieldMeta()` and `deriveUpdatedAt()` from `$lib/data/sync`.
|
||||
- **Origin-tracking**: `originFromActor(actor)` in `@mana/shared-ai` maps `actor.kind` onto `'user' | 'agent' | 'system' | 'migration' | 'server-replay'`. The conflict surface fires only when `localFieldMeta.origin === 'user'` — replay-deltas from server pulls, agent writes, migration helpers, and bootstrap inserts never surface as toasts.
|
||||
- **`updatedAt` is no longer a synced data field.** Type-converters compute `updatedAt` on read as `max(__fieldMeta[*].at)` via `deriveUpdatedAt(local)`. For Dexie-indexed sort, every record carries a non-synced `_updatedAtIndex` shadow column that the hook stamps automatically — `orderBy('_updatedAtIndex')` instead of `orderBy('updatedAt')`.
|
||||
- **Server-side singleton bootstrap**: mana-auth's `/register` flow writes the `userContext` singleton straight into `mana_sync.sync_changes` with `origin: 'system'`. The webapp's `getOrCreateLocalDoc()` survives only as a fallback for the rare race where the first pull hasn't landed yet.
|
||||
- **Stable `client_id`**: Dexie table `_clientIdentity` (single row keyed by `id='self'`) is the canonical source of the per-device sync identity. `restoreClientIdFromDexie()` runs once at boot and reconciles localStorage ↔ Dexie — a localStorage wipe gets restored from Dexie, the server keeps seeing the same client.
|
||||
|
||||
When writing new code:
|
||||
|
||||
| Pattern | Use this |
|
||||
| --- | --- |
|
||||
| Read "last modified" for UI | `deriveUpdatedAt(local)` (returns ISO string) |
|
||||
| Sort a Dexie query by recency | `.orderBy('_updatedAtIndex')` |
|
||||
| Stamp a system/migration write | wrap in `runAsAsync(makeSystemActor(SYSTEM_MIGRATION, '<label>'), async () => { … })` |
|
||||
| Schreib Local-Type | omit `updatedAt: string` field — it's derived, not stored |
|
||||
| Add a new singleton | Bootstrap server-side in mana-auth (see `services/mana-auth/src/services/bootstrap-singletons.ts`) instead of `ensureDoc()` on the client |
|
||||
|
||||
Plan with full per-phase rationale: [`docs/plans/sync-field-meta-overhaul.md`](../../docs/plans/sync-field-meta-overhaul.md).
|
||||
|
||||
## At-Rest Encryption
|
||||
|
||||
User-typed content in **27 tables** is encrypted with **AES-GCM-256** before it touches IndexedDB. Master key lives in `mana-auth` (KEK-wrapped) and is fetched on login.
|
||||
|
|
|
|||
|
|
@ -185,14 +185,35 @@ Logout / Tab-Close → MemoryKeyProvider.setKey(null) → Cyphertext bleibt
|
|||
### Eckdaten
|
||||
|
||||
- **120+ Collections** in einer einzigen IndexedDB
|
||||
- **Schema-Versionen** 1–10 (V9 fügte updatedAt-Indexes hinzu, V10 die `_activity` Tabelle)
|
||||
- **Schema-Versionen** 1–54 (v53 ersetzte `updatedAt`-Indizes durch `_updatedAtIndex`, v54 fügte `_clientIdentity` für stabile Client-IDs hinzu)
|
||||
- **Eager Apps**: mana, todo, calendar, contacts, tags, links — syncen beim Start
|
||||
- **Lazy Apps**: starten Sync erst beim ersten Modul-Besuch via `ensureAppSynced()`
|
||||
- **Conflict Resolution**: ✅ echter Field-Level LWW via `__fieldTimestamps`
|
||||
- **Conflict Resolution**: ✅ Origin-gated Field-Level LWW via `__fieldMeta` (siehe Sync Field-Meta Overhaul unten)
|
||||
- **Soft Delete** ist Standard via `deletedAt`
|
||||
- **At-Rest-Encryption**: AES-GCM-256, server-wrapped Master Key, 27 Tabellen aktiv (Rollout abgeschlossen)
|
||||
- **Zero-Knowledge-Modus** (Phase 9 Opt-In): Recovery-Code-basiertes Wrapping ohne Server-seitige Entschlüsselbarkeit, mit Rotate-im-Active-State-Support
|
||||
|
||||
### Sync Field-Meta Overhaul (2026-04-26, F1–F7 SHIPPED)
|
||||
|
||||
Sieben Phasen, die vier strukturelle Bugs in der Conflict-Detection abgeräumt haben:
|
||||
|
||||
- **F1** (`7766ea502`) `__fieldMeta` ersetzt `__fieldTimestamps` + `__fieldActors` + `__lastActor`. Ein Hidden-Field statt drei. Wire-Format `FieldChange = { value, at }` (vorher `{ value, updatedAt }`). mana-sync DB-Spalte `field_timestamps` → `field_meta`, neue Spalte `origin TEXT`.
|
||||
- **F2** (`ad5e04a55`) Conflict-Trigger gated auf `localFieldMeta.origin === 'user' && !options.isInitialHydration`. `originFromActor()` in shared-ai mappt actor.kind → `'user' | 'agent' | 'system' | 'migration'`. applyServerChanges stempelt alle Replays als `'server-replay'`. Initial-Pull-Hydration suppress alle Conflicts (Belt-and-Suspenders).
|
||||
- **F3** (`6bb9d77be`) `updatedAt` raus aus dem Wire. Read-side computed via `deriveUpdatedAt(record) = max(__fieldMeta[*].at)`. Lokale `_updatedAtIndex`-Schatten-Spalte für Dexie-`orderBy`-Sortierung. v53 Dexie-Migration kopiert `updatedAt` → `_updatedAtIndex` für existing rows. ~382 stamping-sites über 121 Files entfernt.
|
||||
- **F4** (`c07db300b`) Server-side Singleton-Bootstrap in mana-auth's `/register`. `bootstrapUserSingletons(userId)` schreibt `userContext` direkt in `mana_sync.sync_changes` mit `client_id='system:bootstrap'` + `origin='system'`. Default-Inhalt mirror't `emptyUserContext()`.
|
||||
- **F5** (`d78f57c04`) `userContextStore.ensureDoc()` Public-API entfernt. Internal `getOrCreateLocalDoc()` bleibt als Fallback für brand-new clients deren First-Pull noch nicht durch ist. UI mountet ohne ensureDoc-Race.
|
||||
- **F6** (`a031493fe`) Stable `client_id` in Dexie-Tabelle `_clientIdentity`. `restoreClientIdFromDexie()` läuft im (app)-Layout vor `createUnifiedSync` und reconciliated Dexie ↔ localStorage. Dexie ist canonical, localStorage ist fast-read-cache. Survives clear-site-data und incognito flush.
|
||||
- **F7** (`2a8e8ff98`) `repair-silent-twin.ts` + `legacy-avatar.ts` Migrationen ersatzlos gelöscht — pre-live, keine Live-Daten brauchen sie. Orphan-localStorage-Flags-Sweep im Boot (`migrations-cleanup.ts`, `119cd2cf8`) räumt die zugehörigen Flags auf.
|
||||
|
||||
Die vier Bug-Wurzeln (siehe ursprüngliche Diagnose 2026-04-26):
|
||||
|
||||
1. ❌ `updatedAt` als syncbares Feld → entfernt in F3
|
||||
2. ❌ Conflict-Detection unterscheidet User-Edit nicht von Replay → Origin-Gate in F2
|
||||
3. ❌ Singleton race-anfällig → Server-Bootstrap (F4) + Public-API-Drop (F5)
|
||||
4. ❌ `client_id` an localStorage gebunden → Dexie-canonical (F6)
|
||||
|
||||
Plan + Implementation-Notes: [`docs/plans/sync-field-meta-overhaul.md`](../../../../../../docs/plans/sync-field-meta-overhaul.md).
|
||||
|
||||
---
|
||||
|
||||
## 2. Behobene Schwachstellen
|
||||
|
|
@ -220,15 +241,15 @@ Logout / Tab-Close → MemoryKeyProvider.setKey(null) → Cyphertext bleibt
|
|||
|
||||
### 🟢 Mittel — alle erledigt oder in Backlog
|
||||
|
||||
| # | Problem | Status |
|
||||
| --- | ---------------------------------------- | -------------------------------------------------------- |
|
||||
| 12 | `changes: any[]` in `applyServerChanges` | ✅ Sprint 3.1 — `SyncChange` Interface |
|
||||
| 13 | SSE-Buffer akkumuliert ganze Events | ✅ Backlog 2 — pipelined Reader/Apply mit indexOf-Parser |
|
||||
| 14 | Tombstone-Cleanup-Routine fehlt | ✅ Sprint 4+ — `data-layer-listeners.ts` boot+24h Loop |
|
||||
| 15 | Conflict-Visualisierung im UI | 🟢 Backlog — UX-Entscheidung |
|
||||
| 16 | Keine Unit-Tests für `sync.ts` | ✅ Sprint 3.3 — 20 Tests gegen fake-indexeddb |
|
||||
| 17 | Composite Keys / Multi-Account Indizes | 🟢 Backlog — wenn Multi-Account aktiv wird |
|
||||
| 18 | Audit-Log / Activity Feed | ✅ Backlog 3 — `_activity` Table + read API + prune |
|
||||
| # | Problem | Status |
|
||||
| --- | ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 12 | `changes: any[]` in `applyServerChanges` | ✅ Sprint 3.1 — `SyncChange` Interface |
|
||||
| 13 | SSE-Buffer akkumuliert ganze Events | ✅ Backlog 2 — pipelined Reader/Apply mit indexOf-Parser |
|
||||
| 14 | Tombstone-Cleanup-Routine fehlt | ✅ Sprint 4+ — `data-layer-listeners.ts` boot+24h Loop |
|
||||
| 15 | Conflict-Visualisierung im UI | ✅ Sprint 4+ Backlog C — Toast + Restore (`SyncConflictToast`) shipped, F2 (2026-04-26) gated den Trigger auf `origin === 'user'` damit nur echte User-Edits surface |
|
||||
| 16 | Keine Unit-Tests für `sync.ts` | ✅ Sprint 3.3 — 20 Tests gegen fake-indexeddb |
|
||||
| 17 | Composite Keys / Multi-Account Indizes | 🟢 Backlog — wenn Multi-Account aktiv wird |
|
||||
| 18 | Audit-Log / Activity Feed | ✅ Backlog 3 — `_activity` Table + read API + prune |
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue