feat(questions): deep-research module — mana-search + mana-llm pipeline

End-to-end deep-research feature for the questions module: a fire-and-
forget orchestrator in apps/api that plans sub-queries with mana-llm,
retrieves sources via mana-search (with optional Readability extraction),
and streams a structured synthesis back to the web app over SSE.

Backend (apps/api/src/modules/research):
- schema.ts: pgSchema('research') with research_results + sources
- orchestrator.ts: three-phase pipeline (plan / retrieve / synthesise)
  with depth-aware config (quick=1×, standard=3×, deep=6× sub-queries)
- pubsub.ts: in-process event bus, single-node, swappable for Redis
- routes.ts: POST /start (202, fire-and-forget), GET /:id/stream (SSE),
  POST /start-sync (test only), GET /:id, GET /:id/sources
- Credit gating via @mana/shared-hono/credits — validate up-front,
  consume best-effort on `done`. Failed runs cost nothing.

Helpers (apps/api/src/lib):
- llm.ts: llmJson() + llmStream() over mana-llm OpenAI-compat API
- search.ts: webSearch() + bulkExtract() over mana-search Go service
- responses.ts: shared errorResponse / listResponse / validationError

Schema deployment:
- drizzle.config.ts (research-scoped) + drizzle/research/0000_init.sql
  hand-authored migration, deployable via psql -f or drizzle-kit push.
- drizzle-kit added as devDep with db:generate / db:push scripts.

Web client (apps/mana/apps/web/src/lib/api/research.ts):
- Typed start() / get() / listSources() / streamProgress(). The stream
  uses fetch + ReadableStream (not EventSource) so we can attach the
  JWT via Authorization header. Special-cases 402 for friendly toast.
- New PUBLIC_MANA_API_URL plumbing in hooks.server.ts + config.ts.

Module store (modules/questions/stores/answers.svelte.ts):
- New write-side store with createManual / startResearch / accept /
  softDelete. startResearch creates an optimistic empty answer, opens
  the SSE stream, debounces token deltas in 100ms batches into the
  encrypted local row, and on `done` replaces the streamed text with
  the parsed { summary, keyPoints, followUps } payload + citations
  resolved against research.sources.id.

Citation rendering (modules/questions/components/AnswerCitations.svelte):
- Tokenises [n] markers in the answer body into clickable pills with
  hover popovers showing title / host / snippet / external link.
- Lazy-loaded via a session-scoped source cache (stores/sources.svelte.ts)
  that deduplicates concurrent fetches.

UI (routes/(app)/questions/[id]/+page.svelte):
- Recherche card with three-state button (start / cancel / re-run),
  animated phase indicator, source counter.
- Confirmation dialog warning about web/LLM transmission since the
  question itself is locally encrypted.
- Toasts for success / error / cancel via @mana/shared-ui/toast.
- Re-run flow soft-deletes prior research-driven answers but keeps
  manual ones intact.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Till JS 2026-04-08 22:15:35 +02:00
parent 30787e36d2
commit e82851985b
18 changed files with 2221 additions and 4 deletions

View file

@ -0,0 +1,64 @@
/**
* In-process pubsub for research progress events.
*
* Single-node only keeps a Map<researchResultId, Set<subscriber>> in
* memory. When apps/api scales horizontally, swap this for a Redis Pub/Sub
* implementation behind the same publish/subscribe interface.
*
* Subscribers are kept until either the pipeline emits a terminal event
* (`done` / `error`) or the consumer unsubscribes (e.g. SSE client closed).
* Late subscribers do NOT receive backfilled events the routes layer is
* expected to read the current DB state once before subscribing.
*/
import type { ProgressEvent } from './orchestrator';
type Subscriber = (event: ProgressEvent) => void;
const channels = new Map<string, Set<Subscriber>>();
/**
* Publish an event to all current subscribers of `researchResultId`.
* Subscriber callbacks are wrapped in try/catch so a single misbehaving
* listener cannot block the orchestrator's progress.
*/
export function publish(researchResultId: string, event: ProgressEvent): void {
const subs = channels.get(researchResultId);
if (!subs) return;
for (const sub of subs) {
try {
sub(event);
} catch (err) {
console.error(`[research:pubsub] subscriber threw on ${event.type}:`, err);
}
}
}
/**
* Subscribe to events for `researchResultId`. Returns an unsubscribe fn.
* The channel is GC'd once the last subscriber leaves.
*/
export function subscribe(researchResultId: string, fn: Subscriber): () => void {
let subs = channels.get(researchResultId);
if (!subs) {
subs = new Set();
channels.set(researchResultId, subs);
}
subs.add(fn);
return () => {
const current = channels.get(researchResultId);
if (!current) return;
current.delete(fn);
if (current.size === 0) channels.delete(researchResultId);
};
}
/**
* Drop a channel entirely. Called by the orchestrator wrapper after a
* terminal event has been published, so any leftover subscribers (e.g. a
* lingering SSE connection that hasn't ticked yet) get cleaned up.
*/
export function closeChannel(researchResultId: string): void {
channels.delete(researchResultId);
}