From 3b5d58ecbebdc9f3e581c2de30b7e0674b0f7205 Mon Sep 17 00:00:00 2001 From: Till JS Date: Thu, 9 Apr 2026 01:51:20 +0200 Subject: [PATCH] =?UTF-8?q?feat(shared-llm):=20Phase=204=20=E2=80=94=20per?= =?UTF-8?q?sistent=20LLM=20task=20queue?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Until now, modules wanting to use the orchestrator had to await each LLM call inline in their store code. That's fine for foreground tasks ("user clicked summarize") but a non-starter for background work ("auto-tag every new note", "generate a title for every voice memo after STT finishes"). Background tasks need to: - Queue up while no LLM tier is ready, then drain when one becomes available (e.g. user just enabled the browser tier from settings) - Survive page reloads, browser restarts, and the user navigating away mid-execution - Run one at a time without blocking the foreground UI - Allow modules to subscribe to results reactively without polling - Retry transient failures (network, model loading) but not semantic ones (tier-too-low, content blocked) Phase 4 ships exactly that. Architecture: packages/shared-llm/src/queue.ts — LlmTaskQueue class + QueuedTask interface (the persistent row shape) + EnqueueOptions (refType/refId/priority/maxAttempts) + TaskRegistry type (name → LlmTask map) + LlmTaskQueueOptions (table + orchestrator + registry + retryBackoffMs + idleWakeupMs) Public API: - enqueue(task, input, opts) → string (returns the queued id) - get(id), list(filter) - retry(id), cancel(id), purge(olderThanMs) - start(), stop() (idempotent processor lifecycle) apps/mana/apps/web/src/lib/llm-queue.ts — web app singleton - Dedicated `mana-llm-queue` Dexie database (separate from the main `mana` IDB; see comment for the rationale: ephemeral per-device state, no encryption needed, no sync needed, doesn't belong in the long-frozen `mana` schema) - Wires up the queue with llmOrchestrator + taskRegistry - Exposes startLlmQueue() / stopLlmQueue() for the layout hook apps/mana/apps/web/src/lib/llm-task-registry.ts - Maps task names → task objects so the queue processor can look up the implementation when pulling rows off the table. Closures can't be persisted, so we round-trip via name. - Currently registers extractDateTask + summarizeTextTask; module-side tasks land here as we add them. apps/mana/apps/web/src/routes/(app)/+layout.svelte - startLlmQueue() in handleAuthReady's Phase A (auth-independent) so guests + authenticated users both get the queue - stopLlmQueue() in onDestroy as a fire-and-forget cleanup Processor loop semantics (the heart of the implementation): 1. On start(), reclaim any 'running' rows from a crashed previous session — reset them to 'pending'. The orphan recovery is the reason a crash mid-task doesn't leave the queue stuck. 2. findNextRunnable() picks the highest-priority pending task whose `notBefore` (retry-backoff timestamp) is in the past. Sort key: priority desc, then enqueuedAt asc (FIFO within priority). 3. Mark the task running, increment attempts, look up the LlmTask in the registry, hand it to orchestrator.run(). 4. On success: mark done, store result + source + finishedAt. 5. On error: - TierTooLowError or ProviderBlockedError → fail immediately, no retry. These are not transient — the user's settings or the content itself need to change. - Anything else → if attempts < maxAttempts, reset to pending with notBefore = now + retryBackoffMs (default 60s). Else mark failed. 6. When no work is pending, sleep on a Promise that resolves when either (a) someone calls enqueue() (which fires notifyWakeup), or (b) idleWakeupMs elapses (default 30s, safety net for any missed wakeup signal). Module-side reactive reads use Dexie liveQuery directly on the queue table — no special subscription API on the queue itself. This is consistent with how every other Mana module reads its data, so the mental model stays uniform: const tags = useLiveQuery( () => llmQueueDb.tasks .where({ refType: 'note', refId, taskName: 'common.extractTags' }) .reverse().first(), [refId] ); Smoke test: a new "Queue" tab in /llm-test lets you enqueue the existing extractDate / summarize tasks and watch the live state of the queue table via liveQuery. The display includes per-row state badge (pending/running/done/failed), tier source, attempt count, input/output, and a "Done/failed löschen" button that exercises purge(). Co-Authored-By: Claude Opus 4.6 (1M context) --- apps/mana/apps/web/src/lib/llm-queue.ts | 66 ++++ .../apps/web/src/lib/llm-task-registry.ts | 27 ++ .../apps/web/src/routes/(app)/+layout.svelte | 11 + .../src/routes/(app)/llm-test/+page.svelte | 126 ++++++- packages/shared-llm/package.json | 1 + packages/shared-llm/src/index.ts | 10 + packages/shared-llm/src/queue.ts | 322 ++++++++++++++++++ pnpm-lock.yaml | 255 +------------- 8 files changed, 567 insertions(+), 251 deletions(-) create mode 100644 apps/mana/apps/web/src/lib/llm-queue.ts create mode 100644 apps/mana/apps/web/src/lib/llm-task-registry.ts create mode 100644 packages/shared-llm/src/queue.ts diff --git a/apps/mana/apps/web/src/lib/llm-queue.ts b/apps/mana/apps/web/src/lib/llm-queue.ts new file mode 100644 index 000000000..2b623ab79 --- /dev/null +++ b/apps/mana/apps/web/src/lib/llm-queue.ts @@ -0,0 +1,66 @@ +/** + * LLM task queue singleton for the Mana web app. + * + * Wires up @mana/shared-llm's LlmTaskQueue with: + * + * - A dedicated Dexie database (`mana-llm-queue`) — separate from + * the main `mana` IndexedDB. The queue holds ephemeral, per-device + * state that does NOT need encryption (the inputs are user content + * they already see), does NOT need cross-device sync (running on + * device A doesn't help device B), and does NOT belong in the + * long-frozen `mana` schema with its 120+ collections. A separate + * small DB is the right granularity here. + * + * - The shared LlmOrchestrator singleton from @mana/shared-llm. + * + * - The task registry from $lib/llm-task-registry.ts — every task + * name that the queue might encounter has to be listed there + * so the processor can look up the LlmTask object at execution + * time. (Closures can't be persisted, so we round-trip via name.) + * + * The queue is started from the (app)/+layout.svelte's onMount so it + * runs once per page session as long as the app is open. + */ + +import Dexie, { type Table } from 'dexie'; +import { LlmTaskQueue, llmOrchestrator, type QueuedTask } from '@mana/shared-llm'; +import { taskRegistry } from './llm-task-registry'; + +class LlmQueueDb extends Dexie { + tasks!: Table; + + constructor() { + super('mana-llm-queue'); + this.version(1).stores({ + // Indexes: + // id primary key (uuid string) + // state filter on pending/running/done/failed + // refType+refId compound index for module reactive reads + // ("show me all tasks for note X") + // taskName filter by task type + // enqueuedAt sort key for FIFO ordering + tasks: 'id, state, [refType+refId], taskName, enqueuedAt', + }); + } +} + +export const llmQueueDb = new LlmQueueDb(); + +export const llmTaskQueue = new LlmTaskQueue({ + table: llmQueueDb.tasks, + orchestrator: llmOrchestrator, + registry: taskRegistry, +}); + +/** Start the background processor. Idempotent — safe to call from + * layout onMount even if multiple components mount in parallel. */ +export function startLlmQueue(): void { + if (typeof window === 'undefined') return; + llmTaskQueue.start(); +} + +/** Stop the queue and wait for the current task to finish. Used by + * tests and by the layout's onDestroy hook. */ +export async function stopLlmQueue(): Promise { + await llmTaskQueue.stop(); +} diff --git a/apps/mana/apps/web/src/lib/llm-task-registry.ts b/apps/mana/apps/web/src/lib/llm-task-registry.ts new file mode 100644 index 000000000..ebdb7d1ae --- /dev/null +++ b/apps/mana/apps/web/src/lib/llm-task-registry.ts @@ -0,0 +1,27 @@ +/** + * Central registry of all LlmTasks the Mana web app knows about. + * + * The persistent task queue stores task NAMES (strings) rather than + * task OBJECTS, because closures can't be serialised to IndexedDB. + * When the queue processor pulls a row off the table, it looks up + * the task name in this registry to recover the actual LlmTask + * object with its runLlm() / runRules() implementations. + * + * Adding a new task: import it here and add it to the map. The + * convention is `{module}.{action}` for the task name, matching + * the `name` field on the LlmTask itself. + * + * If you forget to register a task, the queue will mark any enqueued + * row with that name as failed with the error + * "Task '' is not registered" — which is at least loud and + * obvious enough to catch the typo immediately. + */ + +import type { TaskRegistry } from '@mana/shared-llm'; +import { extractDateTask } from './llm-tasks/extract-date'; +import { summarizeTextTask } from './llm-tasks/summarize'; + +export const taskRegistry: TaskRegistry = { + [extractDateTask.name]: extractDateTask, + [summarizeTextTask.name]: summarizeTextTask, +}; diff --git a/apps/mana/apps/web/src/routes/(app)/+layout.svelte b/apps/mana/apps/web/src/routes/(app)/+layout.svelte index 2bb7aa9cf..9adf8b370 100644 --- a/apps/mana/apps/web/src/routes/(app)/+layout.svelte +++ b/apps/mana/apps/web/src/routes/(app)/+layout.svelte @@ -36,6 +36,7 @@ import { tagLocalStore, tagMutations, useAllTags } from '@mana/shared-stores'; import { linkLocalStore, linkMutations } from '@mana/shared-links'; import { manaStore } from '$lib/data/local-store'; + import { startLlmQueue, stopLlmQueue } from '$lib/llm-queue'; import { createUnifiedSync } from '$lib/data/sync'; import { networkStore } from '$lib/stores/network.svelte'; import { db } from '$lib/data/database'; @@ -305,6 +306,12 @@ initSharedUload(); await dashboardStore.initialize(); + // Start the persistent LLM task queue. Idempotent — safe to call + // repeatedly. The queue picks up any tasks left in 'pending' state + // from previous sessions (and reclaims orphaned 'running' rows + // from a crashed session) before going idle. See $lib/llm-queue.ts. + startLlmQueue(); + // Restore nav collapsed state if (typeof localStorage !== 'undefined') { const savedCollapsed = localStorage.getItem(STORAGE_KEYS.NAV_COLLAPSED); @@ -373,6 +380,10 @@ unifiedSync?.stopAll(); reminderScheduler.stop(); guestMode?.destroy(); + // Fire-and-forget — we don't need to await; the in-flight task + // will finish in the background and the next page session will + // pick up where we left off. + void stopLlmQueue(); }); // ── Search / Spotlight ─────────────────────────────────── diff --git a/apps/mana/apps/web/src/routes/(app)/llm-test/+page.svelte b/apps/mana/apps/web/src/routes/(app)/llm-test/+page.svelte index 83ac390fe..91e76f50e 100644 --- a/apps/mana/apps/web/src/routes/(app)/llm-test/+page.svelte +++ b/apps/mana/apps/web/src/routes/(app)/llm-test/+page.svelte @@ -21,6 +21,8 @@ } from '@mana/shared-llm'; import { extractDateTask } from '$lib/llm-tasks/extract-date'; import { summarizeTextTask } from '$lib/llm-tasks/summarize'; + import { llmTaskQueue, llmQueueDb } from '$lib/llm-queue'; + import { useLiveQueryWithDefault } from '@mana/local-store/svelte'; import { marked } from 'marked'; import { Robot, Trash, PaperPlaneRight, ClockCounterClockwise } from '@mana/shared-icons'; @@ -53,9 +55,21 @@ // --- State --- let selectedModel: ModelKey = $state('gemma-4-e2b'); - let activeTab: 'chat' | 'extract' | 'classify' | 'compare' | 'benchmark' | 'router' = + let activeTab: 'chat' | 'extract' | 'classify' | 'compare' | 'benchmark' | 'router' | 'queue' = $state('chat'); + // --- Queue tab state --- + let queueInput = $state('Treffen mit Sara morgen 14:30'); + let queueLastEnqueuedId = $state(null); + const queueRows = useLiveQueryWithDefault( + async () => llmQueueDb.tasks.orderBy('enqueuedAt').reverse().limit(20).toArray(), + [] + ); + + async function enqueueTaskNow(task: typeof extractDateTask | typeof summarizeTextTask) { + queueLastEnqueuedId = await llmTaskQueue.enqueue(task, { text: queueInput }); + } + // --- Router tab state --- const settings = $derived(llmSettingsState.current); let routerInput = $state('Treffen mit Sara morgen 14:30'); @@ -631,7 +645,7 @@
- {#each [{ id: 'chat', label: 'Chat' }, { id: 'extract', label: 'JSON Extract' }, { id: 'classify', label: 'Classify' }, { id: 'compare', label: 'Compare' }, { id: 'benchmark', label: 'Benchmark' }, { id: 'router', label: 'Router' }] as tab} + {#each [{ id: 'chat', label: 'Chat' }, { id: 'extract', label: 'JSON Extract' }, { id: 'classify', label: 'Classify' }, { id: 'compare', label: 'Compare' }, { id: 'benchmark', label: 'Benchmark' }, { id: 'router', label: 'Router' }, { id: 'queue', label: 'Queue' }] as tab}
{/if} + + + {#if activeTab === 'queue'} +
+
+

+ Smoke-Test für die persistente Task-Queue. Tasks werden in einer eigenen Dexie-DB (mana-llm-queue) gespeichert und im Hintergrund vom Queue-Processor abgearbeitet sobald ein passender + LLM-Tier verfügbar ist. Tasks überleben Page-Reloads — du kannst die Seite hart neuladen + und sie laufen weiter. +

+ + + +
+ + +
+ + {#if queueLastEnqueuedId} +
+ Letzte Task-ID: + {queueLastEnqueuedId} +
+ {/if} +
+ + +
+
+

Letzte 20 Tasks

+ +
+ + {#if queueRows.value.length === 0} +
+ Queue ist leer. Reihe oben einen Task ein. +
+ {:else} +
+ {#each queueRows.value as row} + {@const stateColor = + row.state === 'done' + ? 'border-emerald-500/40 bg-emerald-500/5 text-emerald-600 dark:text-emerald-400' + : row.state === 'failed' + ? 'border-red-500/40 bg-red-500/5 text-red-600 dark:text-red-400' + : row.state === 'running' + ? 'border-blue-500/40 bg-blue-500/5 text-blue-600 dark:text-blue-400' + : 'border-muted-foreground/30 bg-muted/10 text-muted-foreground'} +
+
+ + {row.state} + + {row.taskName} + + · {new Date(row.enqueuedAt).toLocaleTimeString()} + + {#if row.attempts > 1} + + · {row.attempts}/{row.maxAttempts} attempts + + {/if} + {#if row.source} + · via {row.source} + {/if} +
+
+ input: {JSON.stringify(row.input)} +
+ {#if row.result !== undefined} +
+ result: {JSON.stringify(row.result)} +
+ {/if} + {#if row.error} +
error: {row.error}
+ {/if} +
+ {/each} +
+ {/if} +
+
+ {/if} {/if} diff --git a/packages/shared-llm/package.json b/packages/shared-llm/package.json index f18c13d18..f8711d415 100644 --- a/packages/shared-llm/package.json +++ b/packages/shared-llm/package.json @@ -21,6 +21,7 @@ "typescript": "^5.9.3" }, "peerDependencies": { + "dexie": "^4.0.0", "svelte": "^5.0.0" } } diff --git a/packages/shared-llm/src/index.ts b/packages/shared-llm/src/index.ts index dc36aedb9..0055d8d5e 100644 --- a/packages/shared-llm/src/index.ts +++ b/packages/shared-llm/src/index.ts @@ -26,6 +26,16 @@ export { // Task contract export { buildTaskRequest, type LlmTask } from './task'; +// Persistent task queue +export { + LlmTaskQueue, + type EnqueueOptions, + type LlmTaskQueueOptions, + type QueuedTask, + type QueuedTaskState, + type TaskRegistry, +} from './queue'; + // Orchestrator (rarely instantiated directly — most consumers use the // store's singleton instead) export { LlmOrchestrator, type LlmOrchestratorOptions } from './orchestrator'; diff --git a/packages/shared-llm/src/queue.ts b/packages/shared-llm/src/queue.ts new file mode 100644 index 000000000..b40f2ac5b --- /dev/null +++ b/packages/shared-llm/src/queue.ts @@ -0,0 +1,322 @@ +/** + * LlmTaskQueue — persistent fire-and-forget LLM task processor. + * + * Modules call `queue.enqueue(task, input, opts)` to schedule an LLM + * task without blocking the UI. The queue persists every entry in a + * Dexie table, so tasks survive page reloads, browser restarts, and + * the user navigating away mid-execution. A background loop picks + * pending tasks one at a time, runs them through the orchestrator + * (which itself decides which tier to use based on user settings), + * and writes the result back to the same row. + * + * Modules read results reactively via Dexie liveQuery on the same + * table — no subscription API needed on the queue itself, the + * standard `useLiveQuery(() => table.where(...))` pattern just works. + * + * Concurrency: ONE task at a time. Browser-tier inference is + * single-threaded (one WebGPU device on one worker), so parallel + * generations are sequential anyway, and the simpler scheduler beats + * a complicated one until we have a real reason to need it. + * + * Failure model: + * - Retries with flat 60s backoff up to maxAttempts (default 3). + * - TierTooLowError and ProviderBlockedError are NOT retried — they + * are not transient. The task is marked failed and the module's + * UI can offer the user a "switch tier" or "retry manually" + * prompt via the standard Dexie reactive read. + * - Network/load errors (BackendUnreachableError, EdgeLoadFailedError) + * ARE retried — they might recover when the user reconnects or + * loads the model. + * - Tasks left in the 'running' state at startup are reclaimed and + * reset to 'pending' (the previous page session presumably crashed + * or was closed mid-execution). + */ + +import type { Table } from 'dexie'; +import type { LlmOrchestrator } from './orchestrator'; +import type { LlmTask } from './task'; +import type { LlmTier } from './tiers'; +import { ProviderBlockedError, TierTooLowError } from './errors'; + +export type QueuedTaskState = 'pending' | 'running' | 'done' | 'failed'; + +/** + * The persistent shape of a task in the queue. The `result` and + * `error` fields are optional and populated only after execution. + * `input` is opaque (`unknown`) — the queue doesn't know or care + * about its shape; it just hands it back to the LlmTask's runLlm / + * runRules implementation. + */ +export interface QueuedTask { + id: string; + taskName: string; + input: unknown; + state: QueuedTaskState; + enqueuedAt: number; + startedAt?: number; + finishedAt?: number; + result?: unknown; + error?: string; + source?: LlmTier; + attempts: number; + maxAttempts: number; + /** Optional module metadata for filtering: 'note', 'todo', etc. */ + refType?: string; + /** Optional module metadata: the entity this task is about. */ + refId?: string; + /** 0 = normal, higher = more urgent. Sort key for the next-pending pick. */ + priority: number; + /** Earliest time this task should run again. Used for retry backoff. */ + notBefore?: number; +} + +export interface EnqueueOptions { + refType?: string; + refId?: string; + priority?: number; + maxAttempts?: number; +} + +/** + * The registry maps task names to task definitions. The queue uses + * it to look up the right LlmTask object when it's time to execute + * a row from the persistent table — the row only stores the task + * NAME (a string), since closures can't be persisted. + * + * The web app builds this registry at startup by importing all of + * its task modules and listing them by name. See + * apps/mana/apps/web/src/lib/llm-task-registry.ts for the canonical + * example. + */ +// eslint-disable-next-line @typescript-eslint/no-explicit-any +export type TaskRegistry = Record>; + +export interface LlmTaskQueueOptions { + table: Table; + orchestrator: LlmOrchestrator; + registry: TaskRegistry; + /** Backoff between retries in ms. Default 60_000 (1 minute). */ + retryBackoffMs?: number; + /** Maximum sleep when no work is pending, ms. Default 30_000. */ + idleWakeupMs?: number; +} + +export class LlmTaskQueue { + private readonly table: Table; + private readonly orchestrator: LlmOrchestrator; + private readonly registry: TaskRegistry; + private readonly retryBackoffMs: number; + private readonly idleWakeupMs: number; + private running = false; + private wakeupResolvers: Array<() => void> = []; + private loopPromise: Promise | null = null; + + constructor(opts: LlmTaskQueueOptions) { + this.table = opts.table; + this.orchestrator = opts.orchestrator; + this.registry = opts.registry; + this.retryBackoffMs = opts.retryBackoffMs ?? 60_000; + this.idleWakeupMs = opts.idleWakeupMs ?? 30_000; + } + + // ─── Public API ────────────────────────────────────────── + + /** Schedule a task for background execution. Returns the queued task id. */ + async enqueue( + task: LlmTask, + input: TIn, + opts: EnqueueOptions = {} + ): Promise { + const id = crypto.randomUUID(); + const queued: QueuedTask = { + id, + taskName: task.name, + input, + state: 'pending', + enqueuedAt: Date.now(), + attempts: 0, + maxAttempts: opts.maxAttempts ?? 3, + refType: opts.refType, + refId: opts.refId, + priority: opts.priority ?? 0, + }; + await this.table.add(queued); + this.notifyWakeup(); + return id; + } + + /** Look up a task by id. */ + async get(id: string): Promise { + return this.table.get(id); + } + + /** Manually retry a failed task. Resets state to pending and clears the error. */ + async retry(id: string): Promise { + await this.table.update(id, { + state: 'pending', + error: undefined, + attempts: 0, + notBefore: undefined, + }); + this.notifyWakeup(); + } + + /** Cancel a task — removes it from the queue if not yet running. */ + async cancel(id: string): Promise { + const task = await this.table.get(id); + if (!task) return; + if (task.state === 'running') { + throw new Error(`Cannot cancel task ${id} — it's currently running`); + } + await this.table.delete(id); + } + + /** Clear all done/failed tasks older than the given timestamp. + * Use this for periodic cleanup so the queue table doesn't grow + * unbounded. */ + async purge(olderThanMs: number): Promise { + const cutoff = Date.now() - olderThanMs; + const stale = await this.table + .filter((t) => (t.state === 'done' || t.state === 'failed') && (t.finishedAt ?? 0) < cutoff) + .toArray(); + const ids = stale.map((t) => t.id); + if (ids.length > 0) await this.table.bulkDelete(ids); + return ids.length; + } + + /** Start the background processor. Idempotent. */ + start(): void { + if (this.running) return; + this.running = true; + this.loopPromise = this.loop(); + } + + /** Stop the background processor. Returns when the current task + * (if any) finishes. */ + async stop(): Promise { + this.running = false; + this.notifyWakeup(); + if (this.loopPromise) { + await this.loopPromise; + this.loopPromise = null; + } + } + + // ─── Internal: processor loop ──────────────────────────── + + private async loop(): Promise { + // On startup, reclaim orphaned 'running' rows from a crashed + // previous session. + await this.reclaimOrphaned(); + + while (this.running) { + const next = await this.findNextRunnable(); + if (!next) { + await this.waitForWakeup(this.idleWakeupMs); + continue; + } + + await this.executeTask(next); + } + } + + private async reclaimOrphaned(): Promise { + const orphans = await this.table.where('state').equals('running').toArray(); + for (const o of orphans) { + await this.table.update(o.id, { state: 'pending', startedAt: undefined }); + } + } + + /** Find the next pending task that's eligible to run. + * Highest priority first, then oldest enqueuedAt first. + * Skips tasks whose `notBefore` is still in the future (retry backoff). */ + private async findNextRunnable(): Promise { + const now = Date.now(); + const pending = await this.table.where('state').equals('pending').toArray(); + const eligible = pending.filter((t) => (t.notBefore ?? 0) <= now); + if (eligible.length === 0) return undefined; + eligible.sort((a, b) => { + if (b.priority !== a.priority) return b.priority - a.priority; + return a.enqueuedAt - b.enqueuedAt; + }); + return eligible[0]; + } + + private async executeTask(task: QueuedTask): Promise { + const definition = this.registry[task.taskName]; + if (!definition) { + // Task name no longer registered (e.g. module was removed). + // Mark as failed permanently. + await this.table.update(task.id, { + state: 'failed', + error: `Task '${task.taskName}' is not registered`, + finishedAt: Date.now(), + }); + return; + } + + // Mark as running + await this.table.update(task.id, { + state: 'running', + startedAt: Date.now(), + attempts: task.attempts + 1, + }); + + try { + const result = await this.orchestrator.run(definition, task.input); + await this.table.update(task.id, { + state: 'done', + result: result.value, + source: result.source, + finishedAt: Date.now(), + error: undefined, + }); + } catch (err) { + const errorMessage = err instanceof Error ? err.message : String(err); + const attempts = task.attempts + 1; + + // Non-retryable errors: tier-too-low (won't change without + // settings change) and provider-blocked (content rejected). + const nonRetryable = err instanceof TierTooLowError || err instanceof ProviderBlockedError; + + if (nonRetryable || attempts >= task.maxAttempts) { + await this.table.update(task.id, { + state: 'failed', + error: errorMessage, + finishedAt: Date.now(), + }); + } else { + // Retry with backoff + await this.table.update(task.id, { + state: 'pending', + error: errorMessage, + notBefore: Date.now() + this.retryBackoffMs, + }); + } + } + } + + // ─── Internal: wakeup signaling ────────────────────────── + + private notifyWakeup(): void { + const resolvers = this.wakeupResolvers; + this.wakeupResolvers = []; + for (const r of resolvers) r(); + } + + private waitForWakeup(timeoutMs: number): Promise { + return new Promise((resolve) => { + let resolved = false; + const finish = () => { + if (resolved) return; + resolved = true; + resolve(); + }; + const timer = setTimeout(finish, timeoutMs); + this.wakeupResolvers.push(() => { + clearTimeout(timer); + finish(); + }); + }); + } +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 28bd3bca9..d7e8f82b3 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -1567,6 +1567,9 @@ importers: apps/memoro/apps/server: dependencies: + '@mana/notify-client': + specifier: workspace:* + version: link:../../../../packages/notify-client '@mana/shared-hono': specifier: workspace:* version: link:../../../../packages/shared-hono @@ -2565,34 +2568,6 @@ importers: specifier: ^2.7.0 version: 2.7.0 - packages/cards-database: - dependencies: - drizzle-orm: - specifier: ^0.36.0 - version: 0.36.4(@opentelemetry/api@1.9.1)(@types/pg@8.6.1)(@types/react@19.2.14)(bun-types@1.3.11)(kysely@0.28.15)(postgres@3.4.9)(react@19.2.0) - postgres: - specifier: ^3.4.5 - version: 3.4.9 - devDependencies: - '@supabase/supabase-js': - specifier: ^2.81.1 - version: 2.102.1 - '@types/node': - specifier: ^22.10.0 - version: 22.19.17 - dotenv-cli: - specifier: ^7.4.0 - version: 7.4.4 - drizzle-kit: - specifier: ^0.28.0 - version: 0.28.1 - tsx: - specifier: ^4.19.0 - version: 4.21.0 - typescript: - specifier: ^5.7.3 - version: 5.9.3 - packages/credits: devDependencies: svelte: @@ -2762,16 +2737,6 @@ importers: specifier: ^4.1.2 version: 4.1.3(@opentelemetry/api@1.9.1)(@types/node@24.12.2)(@vitest/coverage-v8@4.1.3)(@vitest/ui@4.1.3)(jsdom@29.0.2(@noble/hashes@2.0.1))(vite@6.4.2(@types/node@24.12.2)(jiti@2.6.1)(lightningcss@1.32.0)(terser@5.46.1)(tsx@4.21.0)(yaml@2.8.3)) - packages/shared-api-client: - dependencies: - '@mana/shared-utils': - specifier: workspace:* - version: link:../shared-utils - devDependencies: - typescript: - specifier: ^5.9.3 - version: 5.9.3 - packages/shared-auth: dependencies: '@mana/shared-types': @@ -2885,21 +2850,6 @@ importers: specifier: ^5.0.0 version: 5.9.3 - packages/shared-errors: - devDependencies: - '@nestjs/common': - specifier: ^11.0.17 - version: 11.1.18(class-transformer@0.5.1)(class-validator@0.14.4)(reflect-metadata@0.2.2)(rxjs@7.8.2) - '@types/express': - specifier: ^5.0.0 - version: 5.0.6 - '@types/node': - specifier: ^22.0.0 - version: 22.19.17 - typescript: - specifier: ^5.9.3 - version: 5.9.3 - packages/shared-hono: dependencies: '@mana/shared-logger': @@ -2987,6 +2937,9 @@ importers: '@mana/local-llm': specifier: workspace:* version: link:../local-llm + dexie: + specifier: ^4.0.0 + version: 4.4.2 devDependencies: '@types/node': specifier: ^24.10.1 @@ -3022,18 +2975,6 @@ importers: specifier: ^7.0.0 version: 7.4.0(@types/babel__core@7.20.5) - packages/shared-splitscreen: - devDependencies: - svelte: - specifier: ^5.0.0 - version: 5.55.1 - svelte-check: - specifier: ^4.0.0 - version: 4.4.6(picomatch@4.0.4)(svelte@5.55.1)(typescript@5.9.3) - typescript: - specifier: ^5.0.0 - version: 5.9.3 - packages/shared-storage: dependencies: '@aws-sdk/client-s3': @@ -6719,19 +6660,6 @@ packages: class-validator: optional: true - '@nestjs/common@11.1.18': - resolution: {integrity: sha512-0sLq8Z+TIjLnz1Tqp0C/x9BpLbqpt1qEu0VcH4/fkE0y3F5JxhfK1AdKQ/SPbKhKgwqVDoY4gS8GQr2G6ujaWg==} - peerDependencies: - class-transformer: '>=0.4.1' - class-validator: '>=0.13.2' - reflect-metadata: ^0.1.12 || ^0.2.0 - rxjs: ^7.1.0 - peerDependenciesMeta: - class-transformer: - optional: true - class-validator: - optional: true - '@nestjs/config@3.3.0': resolution: {integrity: sha512-pdGTp8m9d0ZCrjTpjkUbZx6gyf2IKf+7zlkrPNMsJzYZ4bFRRTpXrnj+556/5uiI6AfL5mMrJc2u7dB6bvM+VA==} peerDependencies: @@ -8446,10 +8374,6 @@ packages: resolution: {integrity: sha512-MADQgmZT1eKjp06jpI2yozxaU9uVs4GzzgSL+uEq7bVcJ9V1ZXQkeGNql1fsSI0gMy1vhvNTNbUqrx+pZfJVmg==} engines: {node: '>=18'} - '@tokenizer/inflate@0.4.1': - resolution: {integrity: sha512-2mAv+8pkG6GIZiF1kNg1jAjh27IDxEPKwdGul3snfztFerfPGI1LjDezZp3i7BElXompqEtPmoPx6c2wgtWsOA==} - engines: {node: '>=18'} - '@tokenizer/token@0.3.0': resolution: {integrity: sha512-OvjF+z51L3ov0OyAU0duzsYuvO01PH7x4t6DJx+guahgTnBHkhJdG7soQeTSFLWN3efnHyibZ4Z8l2EuWwJN3A==} @@ -10540,10 +10464,6 @@ packages: dot-case@3.0.4: resolution: {integrity: sha512-Kv5nKlh6yRrdrGvxeJ2e5y2eRUpkUosIW4A2AS38zwSz27zu7ufDwQPi5Jhs3XAlGNetl3bmnGhQsMtkKJnj3w==} - dotenv-cli@7.4.4: - resolution: {integrity: sha512-XkBYCG0tPIes+YZr4SpfFv76SQrV/LeCE8CI7JSEMi3VR9MvTihCGTOtbIexD6i2mXF+6px7trb1imVCXSNMDw==} - hasBin: true - dotenv-expand@10.0.0: resolution: {integrity: sha512-GopVGCpVS1UKH75VKHGuQFqS1Gusej0z4FyQkPdwjil2gNIv+LNsqBlboOzpJFZKVT95GkCyWJbBSdFEFUWI2A==} engines: {node: '>=12'} @@ -10564,106 +10484,10 @@ packages: resolution: {integrity: sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==} engines: {node: '>=12'} - drizzle-kit@0.28.1: - resolution: {integrity: sha512-JimOV+ystXTWMgZkLHYHf2w3oS28hxiH1FR0dkmJLc7GHzdGJoJAQtQS5DRppnabsRZwE2U1F6CuezVBgmsBBQ==} - hasBin: true - drizzle-kit@0.30.6: resolution: {integrity: sha512-U4wWit0fyZuGuP7iNmRleQyK2V8wCuv57vf5l3MnG4z4fzNTjY/U13M8owyQ5RavqvqxBifWORaR3wIUzlN64g==} hasBin: true - drizzle-orm@0.36.4: - resolution: {integrity: sha512-1OZY3PXD7BR00Gl61UUOFihslDldfH4NFRH2MbP54Yxi0G/PKn4HfO65JYZ7c16DeP3SpM3Aw+VXVG9j6CRSXA==} - peerDependencies: - '@aws-sdk/client-rds-data': '>=3' - '@cloudflare/workers-types': '>=3' - '@electric-sql/pglite': '>=0.2.0' - '@libsql/client': '>=0.10.0' - '@libsql/client-wasm': '>=0.10.0' - '@neondatabase/serverless': '>=0.10.0' - '@op-engineering/op-sqlite': '>=2' - '@opentelemetry/api': ^1.4.1 - '@planetscale/database': '>=1' - '@prisma/client': '*' - '@tidbcloud/serverless': '*' - '@types/better-sqlite3': '*' - '@types/pg': '*' - '@types/react': '>=18' - '@types/sql.js': '*' - '@vercel/postgres': '>=0.8.0' - '@xata.io/client': '*' - better-sqlite3: '>=7' - bun-types: '*' - expo-sqlite: '>=14.0.0' - knex: '*' - kysely: '*' - mysql2: '>=2' - pg: '>=8' - postgres: '>=3' - prisma: '*' - react: '>=18' - sql.js: '>=1' - sqlite3: '>=5' - peerDependenciesMeta: - '@aws-sdk/client-rds-data': - optional: true - '@cloudflare/workers-types': - optional: true - '@electric-sql/pglite': - optional: true - '@libsql/client': - optional: true - '@libsql/client-wasm': - optional: true - '@neondatabase/serverless': - optional: true - '@op-engineering/op-sqlite': - optional: true - '@opentelemetry/api': - optional: true - '@planetscale/database': - optional: true - '@prisma/client': - optional: true - '@tidbcloud/serverless': - optional: true - '@types/better-sqlite3': - optional: true - '@types/pg': - optional: true - '@types/react': - optional: true - '@types/sql.js': - optional: true - '@vercel/postgres': - optional: true - '@xata.io/client': - optional: true - better-sqlite3: - optional: true - bun-types: - optional: true - expo-sqlite: - optional: true - knex: - optional: true - kysely: - optional: true - mysql2: - optional: true - pg: - optional: true - postgres: - optional: true - prisma: - optional: true - react: - optional: true - sql.js: - optional: true - sqlite3: - optional: true - drizzle-orm@0.38.4: resolution: {integrity: sha512-s7/5BpLKO+WJRHspvpqTydxFob8i1vo2rEx4pY6TGY7QSMuUfWUuzaY0DIpXCkgHOo37BaFC+SJQb99dDUXT3Q==} peerDependencies: @@ -12242,10 +12066,6 @@ packages: resolution: {integrity: sha512-hw9gNZXUfZ02Jo0uafWLaFVPter5/k2rfcrjFJJHX/77xtSDOfJuEFb6oKlFV86FLP1SuyHMW1PSk0U9M5tKkQ==} engines: {node: '>=18'} - file-type@21.3.4: - resolution: {integrity: sha512-Ievi/yy8DS3ygGvT47PjSfdFoX+2isQueoYP1cntFW1JLYAuS4GD7NUPGg4zv2iZfV52uDyk5w5Z0TdpRS6Q1g==} - engines: {node: '>=20'} - filelist@1.0.6: resolution: {integrity: sha512-5giy2PkLYY1cP39p17Ech+2xlpTRL9HLspOfEgm0L6CwBXBTgsK5ou0JtzYuepxkaQ/tvhCFIJ5uXo0OrM2DxA==} @@ -13661,10 +13481,6 @@ packages: resolution: {integrity: sha512-ME4Fb83LgEgwNw96RKNvKV4VTLuXfoKudAmm2lP8Kk87KaMK0/Xrx/aAkMWmT8mDb+3MlFDspfbCs7adjRxA2g==} engines: {node: '>=20.0.0'} - load-esm@1.0.3: - resolution: {integrity: sha512-v5xlu8eHD1+6r8EHTg6hfmO97LN8ugKtiXcy5e6oN72iD2r6u0RPfLl6fxM+7Wnh2ZRq15o0russMst44WauPA==} - engines: {node: '>=13.2.0'} - load-tsconfig@0.2.5: resolution: {integrity: sha512-IXO6OCs9yg8tMKzfPZ1YmheJbZCiEsnBdcB03l0OcfK9prKnJb96siuHCr5Fl37/yo9DnKU+TLpxzTUspw9shg==} engines: {node: ^12.20.0 || ^14.13.1 || >=16.0.0} @@ -22003,21 +21819,6 @@ snapshots: transitivePeerDependencies: - supports-color - '@nestjs/common@11.1.18(class-transformer@0.5.1)(class-validator@0.14.4)(reflect-metadata@0.2.2)(rxjs@7.8.2)': - dependencies: - file-type: 21.3.4 - iterare: 1.2.1 - load-esm: 1.0.3 - reflect-metadata: 0.2.2 - rxjs: 7.8.2 - tslib: 2.8.1 - uid: 2.0.2 - optionalDependencies: - class-transformer: 0.5.1 - class-validator: 0.14.4 - transitivePeerDependencies: - - supports-color - '@nestjs/config@3.3.0(@nestjs/common@10.4.22(class-transformer@0.5.1)(class-validator@0.14.4)(reflect-metadata@0.2.2)(rxjs@7.8.2))(rxjs@7.8.2)': dependencies: '@nestjs/common': 10.4.22(class-transformer@0.5.1)(class-validator@0.14.4)(reflect-metadata@0.2.2)(rxjs@7.8.2) @@ -24780,13 +24581,6 @@ snapshots: transitivePeerDependencies: - supports-color - '@tokenizer/inflate@0.4.1': - dependencies: - debug: 4.4.3 - token-types: 6.1.2 - transitivePeerDependencies: - - supports-color - '@tokenizer/token@0.3.0': {} '@turbo/darwin-64@2.9.4': @@ -27791,13 +27585,6 @@ snapshots: no-case: 3.0.4 tslib: 2.8.1 - dotenv-cli@7.4.4: - dependencies: - cross-spawn: 7.0.6 - dotenv: 16.6.1 - dotenv-expand: 10.0.0 - minimist: 1.2.8 - dotenv-expand@10.0.0: {} dotenv-expand@11.0.7: @@ -27810,15 +27597,6 @@ snapshots: dotenv@16.6.1: {} - drizzle-kit@0.28.1: - dependencies: - '@drizzle-team/brocli': 0.10.2 - '@esbuild-kit/esm-loader': 2.6.5 - esbuild: 0.19.12 - esbuild-register: 3.6.0(esbuild@0.19.12) - transitivePeerDependencies: - - supports-color - drizzle-kit@0.30.6: dependencies: '@drizzle-team/brocli': 0.10.2 @@ -27829,16 +27607,6 @@ snapshots: transitivePeerDependencies: - supports-color - drizzle-orm@0.36.4(@opentelemetry/api@1.9.1)(@types/pg@8.6.1)(@types/react@19.2.14)(bun-types@1.3.11)(kysely@0.28.15)(postgres@3.4.9)(react@19.2.0): - optionalDependencies: - '@opentelemetry/api': 1.9.1 - '@types/pg': 8.6.1 - '@types/react': 19.2.14 - bun-types: 1.3.11 - kysely: 0.28.15 - postgres: 3.4.9 - react: 19.2.0 - drizzle-orm@0.38.4(@opentelemetry/api@1.9.1)(@types/pg@8.6.1)(@types/react@19.2.14)(bun-types@1.3.11)(kysely@0.28.15)(postgres@3.4.9)(react@19.2.0): optionalDependencies: '@opentelemetry/api': 1.9.1 @@ -30643,15 +30411,6 @@ snapshots: transitivePeerDependencies: - supports-color - file-type@21.3.4: - dependencies: - '@tokenizer/inflate': 0.4.1 - strtok3: 10.3.5 - token-types: 6.1.2 - uint8array-extras: 1.5.0 - transitivePeerDependencies: - - supports-color - filelist@1.0.6: dependencies: minimatch: 5.1.9 @@ -32492,8 +32251,6 @@ snapshots: rfdc: 1.4.1 wrap-ansi: 9.0.2 - load-esm@1.0.3: {} - load-tsconfig@0.2.5: {} loader-runner@4.3.1: {}