till/managarten
1
0
Fork 0
mirror of https://github.com/Memo-2023/mana-monorepo.git synced 2026-05-15 00:01:10 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 4

feat(questions): deep-research module — mana-search + mana-llm pipeline

Browse source 
End-to-end deep-research feature for the questions module: a fire-and-
forget orchestrator in apps/api that plans sub-queries with mana-llm,
retrieves sources via mana-search (with optional Readability extraction),
and streams a structured synthesis back to the web app over SSE.

Backend (apps/api/src/modules/research):
- schema.ts: pgSchema('research') with research_results + sources
- orchestrator.ts: three-phase pipeline (plan / retrieve / synthesise)
  with depth-aware config (quick=1×, standard=3×, deep=6× sub-queries)
- pubsub.ts: in-process event bus, single-node, swappable for Redis
- routes.ts: POST /start (202, fire-and-forget), GET /:id/stream (SSE),
  POST /start-sync (test only), GET /:id, GET /:id/sources
- Credit gating via @mana/shared-hono/credits — validate up-front,
  consume best-effort on `done`. Failed runs cost nothing.

Helpers (apps/api/src/lib):
- llm.ts: llmJson() + llmStream() over mana-llm OpenAI-compat API
- search.ts: webSearch() + bulkExtract() over mana-search Go service
- responses.ts: shared errorResponse / listResponse / validationError

Schema deployment:
- drizzle.config.ts (research-scoped) + drizzle/research/0000_init.sql
  hand-authored migration, deployable via psql -f or drizzle-kit push.
- drizzle-kit added as devDep with db:generate / db:push scripts.

Web client (apps/mana/apps/web/src/lib/api/research.ts):
- Typed start() / get() / listSources() / streamProgress(). The stream
  uses fetch + ReadableStream (not EventSource) so we can attach the
  JWT via Authorization header. Special-cases 402 for friendly toast.
- New PUBLIC_MANA_API_URL plumbing in hooks.server.ts + config.ts.

Module store (modules/questions/stores/answers.svelte.ts):
- New write-side store with createManual / startResearch / accept /
  softDelete. startResearch creates an optimistic empty answer, opens
  the SSE stream, debounces token deltas in 100ms batches into the
  encrypted local row, and on `done` replaces the streamed text with
  the parsed { summary, keyPoints, followUps } payload + citations
  resolved against research.sources.id.

Citation rendering (modules/questions/components/AnswerCitations.svelte):
- Tokenises [n] markers in the answer body into clickable pills with
  hover popovers showing title / host / snippet / external link.
- Lazy-loaded via a session-scoped source cache (stores/sources.svelte.ts)
  that deduplicates concurrent fetches.

UI (routes/(app)/questions/[id]/+page.svelte):
- Recherche card with three-state button (start / cancel / re-run),
  animated phase indicator, source counter.
- Confirmation dialog warning about web/LLM transmission since the
  question itself is locally encrypted.
- Toasts for success / error / cancel via @mana/shared-ui/toast.
- Re-run flow soft-deletes prior research-driven answers but keeps
  manual ones intact.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Till JS 2026-04-08 22:15:35 +02:00
parent 30787e36d2
commit e82851985b
18 changed files with 2221 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

20
apps/api/drizzle.config.ts Normal file
View file

@ -0,0 +1,20 @@
import { defineConfig } from 'drizzle-kit';
/**
* Drizzle config for the unified mana-api.
*
* Most modules in apps/api inline their schemas in routes.ts and create
* tables out-of-band (or piggyback on schemas owned by other services).
* This config currently only manages the `research` schema introduced for
* the deep-research feature; expand the `schema` glob and `schemaFilter`
* as more modules adopt managed migrations.
*/
export default defineConfig({
schema: './src/modules/research/schema.ts',
out: './drizzle/research',
dialect: 'postgresql',
dbCredentials: {
url: process.env.DATABASE_URL || 'postgresql://mana:devpassword@localhost:5432/mana_platform',
},
schemaFilter: ['research'],
});

47
apps/api/drizzle/research/0000_init.sql Normal file
View file

@ -0,0 +1,47 @@
-- Research module — initial schema (manually authored to match
-- apps/api/src/modules/research/schema.ts).
--
-- Once `drizzle-kit` is installed in apps/api, future migrations should
-- be generated via `pnpm --filter @mana/api db:generate` and this file
-- can become the canonical baseline.
--
-- Apply with:
-- psql "$DATABASE_URL" -f apps/api/drizzle/research/0000_init.sql
CREATE SCHEMA IF NOT EXISTS "research";
CREATE TABLE IF NOT EXISTS "research"."research_results" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
"user_id" text NOT NULL,
"question_id" text NOT NULL,
"depth" text NOT NULL,
"status" text NOT NULL,
"sub_queries" jsonb,
"summary" text,
"key_points" jsonb,
"follow_up_questions" jsonb,
"error_message" text,
"started_at" timestamptz NOT NULL DEFAULT now(),
"finished_at" timestamptz
);
CREATE INDEX IF NOT EXISTS "research_results_user_id_idx"
ON "research"."research_results" ("user_id");
CREATE INDEX IF NOT EXISTS "research_results_question_id_idx"
ON "research"."research_results" ("question_id");
CREATE TABLE IF NOT EXISTS "research"."sources" (
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid(),
"research_result_id" uuid NOT NULL REFERENCES "research"."research_results"("id") ON DELETE CASCADE,
"url" text NOT NULL,
"title" text,
"snippet" text,
"extracted_content" text,
"category" text,
"rank" integer NOT NULL,
"created_at" timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX IF NOT EXISTS "sources_research_result_id_idx"
ON "research"."sources" ("research_result_id");

5
apps/api/package.json
View file

@ -7,7 +7,9 @@
"dev": "bun --hot run src/index.ts",
"build": "bun build src/index.ts --outdir dist --target bun",
"start": "bun run dist/index.js",
"type-check": "tsc --noEmit"
"type-check": "tsc --noEmit",
"db:generate": "drizzle-kit generate",
"db:push": "drizzle-kit push"
},
"dependencies": {
"@mana/media-client": "workspace:*",
@ -24,6 +26,7 @@
"devDependencies": {
"@types/bun": "latest",
"@types/jsdom": "^21.1.0",
"drizzle-kit": "^0.30.0",
"typescript": "^5.8.0"
}
}

2
apps/api/src/index.ts
View file

@ -31,6 +31,7 @@ import { moodlitRoutes } from './modules/moodlit/routes';
import { newsRoutes } from './modules/news/routes';
import { tracesRoutes } from './modules/traces/routes';
import { presiRoutes } from './modules/presi/routes';
import { researchRoutes } from './modules/research/routes';
const PORT = parseInt(process.env.PORT || '3060', 10);
const CORS_ORIGINS = (process.env.CORS_ORIGINS || 'http://localhost:5173').split(',');
@ -61,6 +62,7 @@ app.route('/api/v1/moodlit', moodlitRoutes);
app.route('/api/v1/news', newsRoutes);
app.route('/api/v1/traces', tracesRoutes);
app.route('/api/v1/presi', presiRoutes);
app.route('/api/v1/research', researchRoutes);
// ─── Server Info ────────────────────────────────────────────
console.log(`mana-api starting on port ${PORT}...`);

175
apps/api/src/lib/llm.ts Normal file
View file

@ -0,0 +1,175 @@
/**
* Thin client for the mana-llm gateway.
*
* Two helpers, deliberately small:
*
* llmJson() — non-streaming, parses the model response as JSON.
* Used for plan/structuring steps where we need a typed object.
*
* llmStream() — streaming, calls onToken() for each delta and returns
* the full concatenated text at the end. Used for synthesis.
*
* mana-llm exposes an OpenAI-compatible /api/v1/chat/completions endpoint
* (see services/mana-llm). Models are namespaced as `provider/model`, e.g.
* `ollama/gemma3:4b`, `openrouter/meta-llama/llama-3.1-70b-instruct`.
*
* Internal service-to-service calls — no auth on the wire (private network).
*/
const LLM_URL = process.env.MANA_LLM_URL || 'http://localhost:3025';
export interface LlmMessage {
role: 'system' | 'user' | 'assistant';
content: string;
}
export interface LlmJsonOptions {
model: string;
system?: string;
user: string;
temperature?: number;
maxTokens?: number;
}
export interface LlmStreamOptions {
model: string;
system?: string;
user: string;
temperature?: number;
maxTokens?: number;
onToken: (delta: string) => void | Promise<void>;
signal?: AbortSignal;
}
export class LlmError extends Error {
constructor(
message: string,
public readonly status?: number,
public readonly body?: string
) {
super(message);
this.name = 'LlmError';
}
}
function buildMessages(system: string | undefined, user: string): LlmMessage[] {
const msgs: LlmMessage[] = [];
if (system) msgs.push({ role: 'system', content: system });
msgs.push({ role: 'user', content: user });
return msgs;
}
/**
* Call the LLM and parse the response as JSON.
*
* Strips markdown code fences if the model wraps its output in ```json ... ```.
* Throws LlmError on transport/HTTP failure or if the body isn't valid JSON.
*/
export async function llmJson<T = unknown>(opts: LlmJsonOptions): Promise<T> {
const res = await fetch(`${LLM_URL}/api/v1/chat/completions`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: opts.model,
messages: buildMessages(opts.system, opts.user),
temperature: opts.temperature ?? 0.2,
max_tokens: opts.maxTokens ?? 1000,
response_format: { type: 'json_object' },
}),
});
if (!res.ok) {
const body = await res.text().catch(() => '');
throw new LlmError(`mana-llm returned ${res.status}`, res.status, body);
}
const data = (await res.json()) as {
choices?: Array<{ message?: { content?: string } }>;
};
const raw = data.choices?.[0]?.message?.content;
if (!raw) throw new LlmError('mana-llm response missing content');
const cleaned = stripCodeFence(raw);
try {
return JSON.parse(cleaned) as T;
} catch (err) {
throw new LlmError(
`mana-llm returned non-JSON content: ${(err as Error).message}`,
undefined,
raw
);
}
}
/**
* Call the LLM in streaming mode. Invokes onToken() for each delta and
* returns the full concatenated text once the stream completes.
*
* Parses OpenAI-style SSE: lines beginning with `data: ` and the
* sentinel `data: [DONE]`.
*/
export async function llmStream(opts: LlmStreamOptions): Promise<string> {
const res = await fetch(`${LLM_URL}/api/v1/chat/completions`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
model: opts.model,
messages: buildMessages(opts.system, opts.user),
temperature: opts.temperature ?? 0.5,
max_tokens: opts.maxTokens ?? 2000,
stream: true,
}),
signal: opts.signal,
});
if (!res.ok || !res.body) {
const body = await res.text().catch(() => '');
throw new LlmError(`mana-llm stream returned ${res.status}`, res.status, body);
}
const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';
let full = '';
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
// SSE frames are separated by blank lines, but mana-llm forwards
// line-by-line — split on \n and keep the last (possibly partial) line.
const lines = buffer.split('\n');
buffer = lines.pop() ?? '';
for (const line of lines) {
if (!line.startsWith('data: ')) continue;
const payload = line.slice(6).trim();
if (!payload || payload === '[DONE]') continue;
try {
const chunk = JSON.parse(payload) as {
choices?: Array<{ delta?: { content?: string } }>;
};
const delta = chunk.choices?.[0]?.delta?.content;
if (delta) {
full += delta;
await opts.onToken(delta);
}
} catch {
// ignore malformed frames — keepalives, comments, etc.
}
}
}
return full;
}
function stripCodeFence(text: string): string {
const trimmed = text.trim();
if (!trimmed.startsWith('```')) return trimmed;
// ```json\n...\n``` or ```\n...\n```
const withoutOpen = trimmed.replace(/^```(?:json)?\s*\n?/, '');
return withoutOpen.replace(/\n?```\s*$/, '');
}

118
apps/api/src/lib/responses.ts Normal file
View file

@ -0,0 +1,118 @@
/**
* Standard response helpers for mana-api modules.
*
* Background: A pre-launch audit (April 2026, see
* `docs/REFACTORING_AUDIT_2026_04.md` item #5) flagged that error and
* list responses were inconsistent across the 15+ modules. The actual
* inconsistency turned out to be smaller than reported — every module
* already returns errors as `{ error: 'message' }` — but using these
* helpers gives us:
*
* 1. **Type-safe status codes** — TS catches stray `c.json(..., 999)`
* 2. **One place to enrich the envelope** — when we add `code`,
* `requestId`, or `details` later, we change one file instead of
* grepping 79 callsites.
* 3. **Consistent list shape** — `{ items, count }` regardless of
* what the items are. Frontend `apps/mana/apps/web` doesn't have
* to special-case `events` vs `contacts` vs `occurrences`.
*
* The shape is wire-compatible with the existing inline `c.json(...)`
* calls, so adoption can be incremental: new code uses these helpers,
* old code keeps working until someone touches the file.
*
* @example
* ```ts
* import { errorResponse, listResponse, validationError } from '../../lib/responses';
*
* routes.get('/things', async (c) => {
* const things = await db.select().from(thingsTable);
* return listResponse(c, things);
* });
*
* routes.post('/things', async (c) => {
* const parsed = thingSchema.safeParse(await c.req.json());
* if (!parsed.success) return validationError(c, parsed.error.issues);
* // ...
* });
* ```
*/
import type { Context } from 'hono';
import type { ContentfulStatusCode } from 'hono/utils/http-status';
/**
* Standard error response envelope.
*
* Wire-compatible with the inline `c.json({ error: '...' }, status)`
* pattern that already dominates the codebase. Future fields like
* `code` (machine-readable error code) and `details` (validation issues,
* etc.) can be added without touching callsites.
*/
export type ErrorBody = {
error: string;
code?: string;
details?: unknown;
};
/**
* Standard list response envelope.
*
* Always uses `items` as the field name, regardless of what's inside.
* The frontend hits a stable shape: `{ items: T[], count: number }`.
*/
export type ListBody<T> = {
items: T[];
count: number;
};
/**
* Return a structured error response.
*
* @param c Hono context
* @param error Human-readable message (also used as fallback for code)
* @param status HTTP status (default 500)
* @param extra Optional extra fields — `code` for machine-readable
* identification, `details` for validation issues, etc.
*/
export function errorResponse(
c: Context,
error: string,
status: ContentfulStatusCode = 500,
extra?: { code?: string; details?: unknown }
) {
const body: ErrorBody = { error };
if (extra?.code) body.code = extra.code;
if (extra?.details !== undefined) body.details = extra.details;
return c.json(body, status);
}
/**
* Return a validation error response (400) with structured issues.
*
* Convenience over `errorResponse` for the common Zod case — extracts
* the first error message as the human string and attaches the full
* issue list under `details`.
*/
export function validationError(c: Context, issues: unknown[], status: ContentfulStatusCode = 400) {
const firstMessage =
Array.isArray(issues) &&
issues.length > 0 &&
typeof issues[0] === 'object' &&
issues[0] !== null &&
'message' in issues[0]
? String((issues[0] as { message: unknown }).message)
: 'Invalid input';
return errorResponse(c, firstMessage, status, { code: 'VALIDATION', details: issues });
}
/**
* Return a standard list response. Always wraps in `{ items, count }`,
* regardless of what `items` are. This is the *opposite* of the current
* convention where each module names its own field
* (`{ events, count }`, `{ contacts, count }`) — frontends benefit
* from a single uniform unwrap step.
*/
export function listResponse<T>(c: Context, items: T[], status: ContentfulStatusCode = 200) {
const body: ListBody<T> = { items, count: items.length };
return c.json(body, status);
}

120
apps/api/src/lib/search.ts Normal file
View file

@ -0,0 +1,120 @@
/**
* Thin client for the mana-search Go service.
*
* Two helpers, scoped to what the research orchestrator needs:
*
* webSearch() — POST /api/v1/search, returns ranked SearXNG results.
* bulkExtract() — POST /api/v1/extract/bulk, returns Readability text per URL.
*
* Internal service-to-service calls — no auth on the wire (private network).
*/
const SEARCH_URL = process.env.MANA_SEARCH_URL || 'http://localhost:3021';
export interface SearchHit {
url: string;
title: string;
snippet: string;
engine: string;
score: number;
publishedDate?: string;
category: string;
}
export interface ExtractedContent {
title: string;
description?: string;
author?: string;
publishedDate?: string;
siteName?: string;
text: string;
wordCount: number;
}
export interface BulkExtractResult {
url: string;
success: boolean;
content?: ExtractedContent;
error?: string;
}
export class SearchError extends Error {
constructor(
message: string,
public readonly status?: number
) {
super(message);
this.name = 'SearchError';
}
}
export interface WebSearchOptions {
query: string;
limit?: number;
categories?: string[]; // 'general' | 'news' | 'science' | 'it'
language?: string;
signal?: AbortSignal;
}
/** Run one SearXNG query via mana-search and return normalised hits. */
export async function webSearch(opts: WebSearchOptions): Promise<SearchHit[]> {
const res = await fetch(`${SEARCH_URL}/api/v1/search`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
query: opts.query,
options: {
limit: opts.limit ?? 10,
categories: opts.categories,
language: opts.language ?? 'de-DE',
},
}),
signal: opts.signal,
});
if (!res.ok) {
throw new SearchError(`mana-search returned ${res.status}`, res.status);
}
const data = (await res.json()) as { results?: SearchHit[] };
return data.results ?? [];
}
/**
* Extract Readability content for a batch of URLs in parallel server-side.
* mana-search caps at 20 URLs per call; we slice if more come in.
*/
export async function bulkExtract(
urls: string[],
opts: { maxLength?: number; concurrency?: number; signal?: AbortSignal } = {}
): Promise<BulkExtractResult[]> {
if (urls.length === 0) return [];
const batches: string[][] = [];
for (let i = 0; i < urls.length; i += 20) batches.push(urls.slice(i, i + 20));
const all: BulkExtractResult[] = [];
for (const batch of batches) {
const res = await fetch(`${SEARCH_URL}/api/v1/extract/bulk`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
urls: batch,
concurrency: opts.concurrency ?? 5,
options: {
maxLength: opts.maxLength ?? 8000,
},
}),
signal: opts.signal,
});
if (!res.ok) {
throw new SearchError(`mana-search bulk extract returned ${res.status}`, res.status);
}
const data = (await res.json()) as { results?: BulkExtractResult[] };
if (data.results) all.push(...data.results);
}
return all;
}

366
apps/api/src/modules/research/orchestrator.ts Normal file
View file

@ -0,0 +1,366 @@
/**
* Research orchestrator — three linear phases:
*
* 1. Plan — mana-llm produces N sub-queries (JSON)
* 2. Retrieve — mana-search runs each sub-query in parallel,
* deduplicates, optionally extracts full text
* 3. Synthesise — mana-llm streams a structured answer (summary,
* key points, follow-ups) over the source corpus
*
* Each phase persists its progress to research_results/sources so a
* caller can either await the whole thing (sync mode) or subscribe to
* progress events (will land in routes.ts via a small in-process pubsub).
*
* Errors flip status='error' and surface errorMessage; they never throw
* past runPipeline() so background invocations don't crash the worker.
*/
import { eq } from 'drizzle-orm';
import { db, researchResults, sources, type ResearchDepth } from './schema';
import { llmJson, llmStream, LlmError } from '../../lib/llm';
import { webSearch, bulkExtract, type SearchHit, SearchError } from '../../lib/search';
// ─── Depth configuration ────────────────────────────────────
interface DepthConfig {
subQueryCount: number;
hitsPerQuery: number;
maxSources: number;
extract: boolean;
categories: string[];
planModel: string;
synthModel: string;
}
const DEPTH_CONFIG: Record<ResearchDepth, DepthConfig> = {
quick: {
subQueryCount: 1,
hitsPerQuery: 5,
maxSources: 5,
extract: false,
categories: ['general'],
planModel: 'ollama/gemma3:4b',
synthModel: 'ollama/gemma3:4b',
},
standard: {
subQueryCount: 3,
hitsPerQuery: 8,
maxSources: 15,
extract: true,
categories: ['general', 'news'],
planModel: 'ollama/gemma3:4b',
synthModel: 'ollama/gemma3:12b',
},
deep: {
subQueryCount: 6,
hitsPerQuery: 8,
maxSources: 30,
extract: true,
categories: ['general', 'news', 'science', 'it'],
planModel: 'ollama/gemma3:12b',
synthModel: 'ollama/gemma3:12b',
},
};
// ─── Progress events (consumed by routes.ts pubsub later) ───
export type ProgressEvent =
| { type: 'status'; status: 'planning' | 'searching' | 'extracting' | 'synthesizing' }
| { type: 'plan'; subQueries: string[] }
| { type: 'sources'; count: number }
| { type: 'token'; delta: string }
| { type: 'done'; researchResultId: string }
| { type: 'error'; message: string };
export type ProgressEmitter = (event: ProgressEvent) => void;
const noop: ProgressEmitter = () => {};
// ─── Pipeline input ─────────────────────────────────────────
export interface PipelineInput {
researchResultId: string;
questionTitle: string;
questionDescription?: string;
depth: ResearchDepth;
}
// ─── Synthesis JSON shape ───────────────────────────────────
interface SynthesisPayload {
summary: string;
keyPoints: string[];
followUps: string[];
}
// ─── Public entrypoint ──────────────────────────────────────
/**
* Run the full pipeline. Resolves once the row is in `done` or `error`
* state. Never throws — all failures are caught and persisted.
*/
export async function runPipeline(
input: PipelineInput,
emit: ProgressEmitter = noop
): Promise<void> {
const cfg = DEPTH_CONFIG[input.depth];
const id = input.researchResultId;
try {
// ─── Phase 1: Plan ─────────────────────────────────
await setStatus(id, 'planning');
emit({ type: 'status', status: 'planning' });
const subQueries = await planSubQueries(input, cfg);
await db.update(researchResults).set({ subQueries }).where(eq(researchResults.id, id));
emit({ type: 'plan', subQueries });
// ─── Phase 2: Retrieve ─────────────────────────────
await setStatus(id, 'searching');
emit({ type: 'status', status: 'searching' });
const hits = await runSearches(subQueries, cfg);
const ranked = dedupeAndRank(hits).slice(0, cfg.maxSources);
let enriched = ranked.map((h) => ({
hit: h,
extractedText: undefined as string | undefined,
}));
if (cfg.extract && ranked.length > 0) {
await setStatus(id, 'extracting');
emit({ type: 'status', status: 'extracting' });
const extracts = await bulkExtract(
ranked.map((h) => h.url),
{ maxLength: 8000 }
);
const byUrl = new Map(extracts.map((e) => [e.url, e]));
enriched = ranked.map((h) => ({
hit: h,
extractedText: byUrl.get(h.url)?.content?.text,
}));
}
// Persist sources with stable rank order so citations [n] map to sources[n-1].
await db.insert(sources).values(
enriched.map((e, idx) => ({
researchResultId: id,
url: e.hit.url,
title: e.hit.title,
snippet: e.hit.snippet,
extractedContent: e.extractedText,
category: e.hit.category,
rank: idx + 1,
}))
);
emit({ type: 'sources', count: enriched.length });
// ─── Phase 3: Synthesise ───────────────────────────
await setStatus(id, 'synthesizing');
emit({ type: 'status', status: 'synthesizing' });
const synthesis = await synthesise(input, enriched, cfg, emit);
await db
.update(researchResults)
.set({
status: 'done',
summary: synthesis.summary,
keyPoints: synthesis.keyPoints,
followUpQuestions: synthesis.followUps,
finishedAt: new Date(),
})
.where(eq(researchResults.id, id));
emit({ type: 'done', researchResultId: id });
} catch (err) {
const message = formatError(err);
console.error(`[research:${id}] pipeline failed:`, err);
await db
.update(researchResults)
.set({ status: 'error', errorMessage: message, finishedAt: new Date() })
.where(eq(researchResults.id, id))
.catch(() => {});
emit({ type: 'error', message });
}
}
// ─── Phase 1: Plan ──────────────────────────────────────────
async function planSubQueries(input: PipelineInput, cfg: DepthConfig): Promise<string[]> {
if (cfg.subQueryCount === 1) {
// Cheap path: skip the LLM round-trip, just use the question itself.
return [input.questionTitle];
}
const system =
'Du planst eine Web-Recherche. Antworte ausschließlich als JSON-Objekt mit dem Schlüssel "subQueries" (Array aus Strings). Kein Fließtext, kein Markdown.';
const user = [
`Frage: ${input.questionTitle}`,
input.questionDescription ? `Kontext: ${input.questionDescription}` : null,
'',
`Erzeuge genau ${cfg.subQueryCount} präzise, sich gegenseitig ergänzende Web-Suchanfragen.`,
'Mische deutsche und englische Anfragen, wenn das die Trefferqualität verbessert.',
'Jede Anfrage soll einen anderen Aspekt der Frage abdecken.',
]
.filter(Boolean)
.join('\n');
const result = await llmJson<{ subQueries?: unknown }>({
model: cfg.planModel,
system,
user,
temperature: 0.3,
maxTokens: 400,
});
const queries = Array.isArray(result.subQueries)
? result.subQueries.filter((q): q is string => typeof q === 'string' && q.trim().length > 0)
: [];
if (queries.length === 0) {
// Fallback: don't fail the whole run because the planner produced garbage.
return [input.questionTitle];
}
return queries.slice(0, cfg.subQueryCount);
}
// ─── Phase 2: Retrieve ──────────────────────────────────────
async function runSearches(queries: string[], cfg: DepthConfig): Promise<SearchHit[]> {
const results = await Promise.allSettled(
queries.map((q) =>
webSearch({
query: q,
limit: cfg.hitsPerQuery,
categories: cfg.categories,
})
)
);
const hits: SearchHit[] = [];
for (const r of results) {
if (r.status === 'fulfilled') hits.push(...r.value);
else console.warn('[research] sub-query failed:', r.reason);
}
return hits;
}
/**
* Deduplicate by URL, keeping the highest-scored hit per URL.
* Sort by score descending so the best sources land at the top of the prompt.
*/
function dedupeAndRank(hits: SearchHit[]): SearchHit[] {
const byUrl = new Map<string, SearchHit>();
for (const h of hits) {
const existing = byUrl.get(h.url);
if (!existing || h.score > existing.score) byUrl.set(h.url, h);
}
return [...byUrl.values()].sort((a, b) => b.score - a.score);
}
// ─── Phase 3: Synthesise ────────────────────────────────────
async function synthesise(
input: PipelineInput,
enriched: Array<{ hit: SearchHit; extractedText?: string }>,
cfg: DepthConfig,
emit: ProgressEmitter
): Promise<SynthesisPayload> {
const context = enriched
.map((e, i) => {
const body = e.extractedText ?? e.hit.snippet ?? '';
return `[${i + 1}] ${e.hit.title}\n${e.hit.url}\n${truncate(body, 2000)}`;
})
.join('\n\n---\n\n');
const system = [
'Du bist ein gründlicher Research-Assistent.',
'Antworte ausschließlich als JSON-Objekt mit dieser exakten Form:',
'{ "summary": string, "keyPoints": string[], "followUps": string[] }',
'',
'Regeln:',
'- summary: 2–4 Absätze auf Deutsch, jeder belegbare Claim bekommt eine Citation [n], die auf die Quellen-Nummer verweist.',
'- keyPoints: 3–6 Stichpunkte, jeweils mit mindestens einer [n]-Citation.',
'- followUps: 2–4 weiterführende Fragen, ohne Citations.',
'- Verwende ausschließlich Informationen aus den bereitgestellten Quellen. Wenn die Quellen die Frage nicht beantworten, sag das im summary.',
'- Kein Markdown, keine Code-Fences, nur reines JSON.',
].join('\n');
const user = [
`Frage: ${input.questionTitle}`,
input.questionDescription ? `Kontext: ${input.questionDescription}` : null,
'',
'Quellen:',
context,
]
.filter(Boolean)
.join('\n');
// We stream tokens to the client for live UI feedback, then parse the
// fully-collected text as JSON. The final structured payload is what
// gets persisted; the live tokens are just visual progress.
const fullText = await llmStream({
model: cfg.synthModel,
system,
user,
temperature: 0.4,
maxTokens: 2000,
onToken: (delta) => emit({ type: 'token', delta }),
});
return parseSynthesis(fullText);
}
function parseSynthesis(raw: string): SynthesisPayload {
const trimmed = stripCodeFence(raw.trim());
let parsed: unknown;
try {
parsed = JSON.parse(trimmed);
} catch {
// Last-ditch fallback: surface the raw text as the summary so the
// user at least sees what the model produced.
return { summary: raw.trim(), keyPoints: [], followUps: [] };
}
const obj = (parsed ?? {}) as Record<string, unknown>;
return {
summary: typeof obj.summary === 'string' ? obj.summary : '',
keyPoints: Array.isArray(obj.keyPoints)
? obj.keyPoints.filter((k): k is string => typeof k === 'string')
: [],
followUps: Array.isArray(obj.followUps)
? obj.followUps.filter((k): k is string => typeof k === 'string')
: [],
};
}
// ─── Helpers ────────────────────────────────────────────────
async function setStatus(
id: string,
status: 'planning' | 'searching' | 'extracting' | 'synthesizing'
): Promise<void> {
await db.update(researchResults).set({ status }).where(eq(researchResults.id, id));
}
function truncate(s: string, max: number): string {
if (s.length <= max) return s;
return s.slice(0, max) + '…';
}
function stripCodeFence(text: string): string {
if (!text.startsWith('```')) return text;
const withoutOpen = text.replace(/^```(?:json)?\s*\n?/, '');
return withoutOpen.replace(/\n?```\s*$/, '');
}
function formatError(err: unknown): string {
if (err instanceof LlmError) return `LLM: ${err.message}`;
if (err instanceof SearchError) return `Search: ${err.message}`;
if (err instanceof Error) return err.message;
return String(err);
}

64
apps/api/src/modules/research/pubsub.ts Normal file
View file

@ -0,0 +1,64 @@
/**
* In-process pubsub for research progress events.
*
* Single-node only — keeps a Map<researchResultId, Set<subscriber>> in
* memory. When apps/api scales horizontally, swap this for a Redis Pub/Sub
* implementation behind the same publish/subscribe interface.
*
* Subscribers are kept until either the pipeline emits a terminal event
* (`done` / `error`) or the consumer unsubscribes (e.g. SSE client closed).
* Late subscribers do NOT receive backfilled events — the routes layer is
* expected to read the current DB state once before subscribing.
*/
import type { ProgressEvent } from './orchestrator';
type Subscriber = (event: ProgressEvent) => void;
const channels = new Map<string, Set<Subscriber>>();
/**
* Publish an event to all current subscribers of `researchResultId`.
* Subscriber callbacks are wrapped in try/catch so a single misbehaving
* listener cannot block the orchestrator's progress.
*/
export function publish(researchResultId: string, event: ProgressEvent): void {
const subs = channels.get(researchResultId);
if (!subs) return;
for (const sub of subs) {
try {
sub(event);
} catch (err) {
console.error(`[research:pubsub] subscriber threw on ${event.type}:`, err);
}
}
}
/**
* Subscribe to events for `researchResultId`. Returns an unsubscribe fn.
* The channel is GC'd once the last subscriber leaves.
*/
export function subscribe(researchResultId: string, fn: Subscriber): () => void {
let subs = channels.get(researchResultId);
if (!subs) {
subs = new Set();
channels.set(researchResultId, subs);
}
subs.add(fn);
return () => {
const current = channels.get(researchResultId);
if (!current) return;
current.delete(fn);
if (current.size === 0) channels.delete(researchResultId);
};
}
/**
* Drop a channel entirely. Called by the orchestrator wrapper after a
* terminal event has been published, so any leftover subscribers (e.g. a
* lingering SSE connection that hasn't ticked yet) get cleaned up.
*/
export function closeChannel(researchResultId: string): void {
channels.delete(researchResultId);
}

302
apps/api/src/modules/research/routes.ts Normal file
View file

@ -0,0 +1,302 @@
/**
* Research module — HTTP routes.
*
* POST /api/v1/research/start — fire-and-forget, returns
* { researchResultId } immediately.
* The pipeline runs in the
* background and emits progress
* events via the in-process pubsub.
* GET /api/v1/research/:id/stream — SSE for live progress. Sends an
* initial `snapshot` event with the
* current DB state, then forwards
* pubsub events until `done`/`error`.
* POST /api/v1/research/start-sync — synchronous variant, blocks
* until the pipeline finishes.
* Kept for end-to-end testing.
* GET /api/v1/research/:id — fetch a research_results row.
* GET /api/v1/research/:id/sources — fetch all sources for a run.
*/
import { Hono } from 'hono';
import { streamSSE } from 'hono/streaming';
import { z } from 'zod';
import { and, asc, eq } from 'drizzle-orm';
import { db, researchResults, sources } from './schema';
import { runPipeline, type ProgressEvent } from './orchestrator';
import { publish, subscribe, closeChannel } from './pubsub';
import { errorResponse, listResponse, validationError } from '../../lib/responses';
import type { AuthVariables } from '@mana/shared-hono';
import { validateCredits, consumeCredits } from '@mana/shared-hono/credits';
import type { ResearchDepth } from './schema';
const routes = new Hono<{ Variables: AuthVariables }>();
const StartSchema = z.object({
questionId: z.string().min(1).max(200),
title: z.string().min(3).max(500),
description: z.string().max(4000).optional(),
depth: z.enum(['quick', 'standard', 'deep']),
});
/**
* Credit cost per research depth. Loosely calibrated against the chat
* module: a single local Ollama call costs 0.1 credits there, so a
* `quick` run (one search + one synthesis pass on Ollama) at 1 credit is
* roughly 10× a single chat completion. `deep` is 15× because it does
* 6 sub-queries × 8 hits + bulk extraction + a long synthesis pass.
*/
const RESEARCH_COST: Record<ResearchDepth, number> = {
quick: 1,
standard: 5,
deep: 15,
};
const RESEARCH_OPERATION = 'AI_RESEARCH';
// ─── POST /start ────────────────────────────────────────────
// Fire-and-forget: returns immediately, pipeline runs in background and
// emits progress through pubsub. Subscribe via GET /:id/stream.
routes.post('/start', async (c) => {
const userId = c.get('userId');
const parsed = StartSchema.safeParse(await c.req.json().catch(() => null));
if (!parsed.success) {
return validationError(c, parsed.error.issues);
}
// Reserve credits up-front. We don't deduct yet — credits are consumed
// only when the pipeline reaches `done`. Failed runs cost nothing,
// which keeps the UX forgiving for transient errors.
const cost = RESEARCH_COST[parsed.data.depth];
const validation = await validateCredits(userId, RESEARCH_OPERATION, cost);
if (!validation.hasCredits) {
return errorResponse(c, 'Insufficient credits', 402, {
code: 'INSUFFICIENT_CREDITS',
details: {
required: cost,
available: validation.availableCredits,
},
});
}
const [row] = await db
.insert(researchResults)
.values({
userId,
questionId: parsed.data.questionId,
depth: parsed.data.depth,
status: 'planning',
})
.returning();
void runPipelineWithPubsub(row.id, userId, parsed.data);
return c.json({ researchResultId: row.id, cost }, 202);
});
/**
* Wraps runPipeline so every event is published to the in-process pubsub,
* credits are consumed only on success, and the channel is closed once a
* terminal event has gone out.
*/
async function runPipelineWithPubsub(
researchResultId: string,
userId: string,
input: z.infer<typeof StartSchema>
): Promise<void> {
let succeeded = false;
const emit = (event: ProgressEvent) => {
if (event.type === 'done') succeeded = true;
publish(researchResultId, event);
};
try {
await runPipeline(
{
researchResultId,
questionTitle: input.title,
questionDescription: input.description,
depth: input.depth,
},
emit
);
// Best-effort consume — log on failure but don't block the user.
// The DB row is already in `done` state at this point.
if (succeeded) {
const cost = RESEARCH_COST[input.depth];
const ok = await consumeCredits(
userId,
RESEARCH_OPERATION,
cost,
`Research (${input.depth}): ${input.title.slice(0, 80)}`,
{ researchResultId, depth: input.depth }
);
if (!ok) {
console.warn(
`[research:${researchResultId}] consumeCredits failed — pipeline already finished, leaving as-is`
);
}
}
} finally {
// Give SSE consumers a tick to flush the terminal event before we
// drop the channel.
setTimeout(() => closeChannel(researchResultId), 1000);
}
}
// ─── GET /:id/stream ────────────────────────────────────────
// SSE: emit current DB snapshot first (so late subscribers don't miss the
// run if they connect after `done`), then forward pubsub events until the
// pipeline reaches a terminal state or the client disconnects.
routes.get('/:id/stream', async (c) => {
const userId = c.get('userId');
const id = c.req.param('id');
const [row] = await db
.select()
.from(researchResults)
.where(and(eq(researchResults.id, id), eq(researchResults.userId, userId)))
.limit(1);
if (!row) return errorResponse(c, 'not found', 404, { code: 'NOT_FOUND' });
return streamSSE(c, async (stream) => {
let closed = false;
const queue: ProgressEvent[] = [];
let resolveNext: (() => void) | null = null;
const wake = () => {
if (resolveNext) {
const r = resolveNext;
resolveNext = null;
r();
}
};
const unsubscribe = subscribe(id, (event) => {
queue.push(event);
wake();
});
stream.onAbort(() => {
closed = true;
unsubscribe();
wake();
});
// 1. Initial snapshot — so a client that reconnects mid-run (or
// after a finished run) immediately sees state.
await stream.writeSSE({
event: 'snapshot',
data: JSON.stringify(row),
});
// If the run is already terminal, just close.
if (row.status === 'done' || row.status === 'error') {
unsubscribe();
return;
}
// 2. Drain the live event queue until terminal.
while (!closed) {
while (queue.length > 0) {
const event = queue.shift()!;
await stream.writeSSE({
event: event.type,
data: JSON.stringify(event),
});
if (event.type === 'done' || event.type === 'error') {
closed = true;
break;
}
}
if (closed) break;
await new Promise<void>((resolve) => {
resolveNext = resolve;
});
}
unsubscribe();
});
});
// ─── POST /start-sync ───────────────────────────────────────
// Synchronous variant, blocks until the pipeline finishes. Useful for
// end-to-end smoke tests against real mana-search + mana-llm.
routes.post('/start-sync', async (c) => {
const userId = c.get('userId');
const parsed = StartSchema.safeParse(await c.req.json().catch(() => null));
if (!parsed.success) {
return validationError(c, parsed.error.issues);
}
const [row] = await db
.insert(researchResults)
.values({
userId,
questionId: parsed.data.questionId,
depth: parsed.data.depth,
status: 'planning',
})
.returning();
await runPipeline({
researchResultId: row.id,
questionTitle: parsed.data.title,
questionDescription: parsed.data.description,
depth: parsed.data.depth,
});
const [finished] = await db
.select()
.from(researchResults)
.where(eq(researchResults.id, row.id))
.limit(1);
return c.json(finished);
});
// ─── GET /:id ───────────────────────────────────────────────
routes.get('/:id', async (c) => {
const userId = c.get('userId');
const id = c.req.param('id');
const [row] = await db
.select()
.from(researchResults)
.where(and(eq(researchResults.id, id), eq(researchResults.userId, userId)))
.limit(1);
if (!row) return errorResponse(c, 'not found', 404, { code: 'NOT_FOUND' });
return c.json(row);
});
// ─── GET /:id/sources ───────────────────────────────────────
routes.get('/:id/sources', async (c) => {
const userId = c.get('userId');
const id = c.req.param('id');
// Ownership check via the parent research_results row.
const [parent] = await db
.select({ id: researchResults.id })
.from(researchResults)
.where(and(eq(researchResults.id, id), eq(researchResults.userId, userId)))
.limit(1);
if (!parent) return errorResponse(c, 'not found', 404, { code: 'NOT_FOUND' });
const rows = await db
.select()
.from(sources)
.where(eq(sources.researchResultId, id))
.orderBy(asc(sources.rank));
return listResponse(c, rows);
});
export { routes as researchRoutes };

73
apps/api/src/modules/research/schema.ts Normal file
View file

@ -0,0 +1,73 @@
/**
* Research module — DB schema (Drizzle / pgSchema 'research')
*
* Server-side store for deep-research runs orchestrated by apps/api.
* Lives in mana_platform under its own pgSchema.
*
* - research_results: one row per research run, holds plan + final synthesis
* - sources: one row per web source consumed by a run
*
* The local-first questions module references research_results.id from
* LocalAnswer.researchResultId; sources are fetched on-demand via the API
* and never mirrored into IndexedDB (they're public web content).
*/
import { drizzle } from 'drizzle-orm/postgres-js';
import postgres from 'postgres';
import { pgSchema, uuid, text, timestamp, integer, jsonb } from 'drizzle-orm/pg-core';
const DATABASE_URL =
process.env.DATABASE_URL ?? 'postgresql://mana:devpassword@localhost:5432/mana_platform';
export const researchSchema = pgSchema('research');
/**
* One row per research run. Created in `planning` state immediately on
* /start, then updated as the orchestrator advances through phases.
*/
export const researchResults = researchSchema.table('research_results', {
id: uuid('id').defaultRandom().primaryKey(),
userId: text('user_id').notNull(),
questionId: text('question_id').notNull(), // mirrors local LocalQuestion.id (UUID)
depth: text('depth').notNull(), // 'quick' | 'standard' | 'deep'
status: text('status').notNull(), // 'planning' | 'searching' | 'extracting' | 'synthesizing' | 'done' | 'error'
subQueries: jsonb('sub_queries').$type<string[]>(),
summary: text('summary'),
keyPoints: jsonb('key_points').$type<string[]>(),
followUpQuestions: jsonb('follow_up_questions').$type<string[]>(),
errorMessage: text('error_message'),
startedAt: timestamp('started_at', { withTimezone: true }).defaultNow().notNull(),
finishedAt: timestamp('finished_at', { withTimezone: true }),
});
/**
* Sources consumed during a research run. Rank reflects ordering in the
* synthesis prompt so citation [n] in the summary maps to sources[n-1].
*/
export const sources = researchSchema.table('sources', {
id: uuid('id').defaultRandom().primaryKey(),
researchResultId: uuid('research_result_id')
.notNull()
.references(() => researchResults.id, { onDelete: 'cascade' }),
url: text('url').notNull(),
title: text('title'),
snippet: text('snippet'),
extractedContent: text('extracted_content'),
category: text('category'),
rank: integer('rank').notNull(),
createdAt: timestamp('created_at', { withTimezone: true }).defaultNow().notNull(),
});
const connection = postgres(DATABASE_URL, { max: 5, idle_timeout: 20 });
export const db = drizzle(connection, { schema: { researchResults, sources } });
export type ResearchResult = typeof researchResults.$inferSelect;
export type Source = typeof sources.$inferSelect;
export type ResearchDepth = 'quick' | 'standard' | 'deep';
export type ResearchStatus =
| 'planning'
| 'searching'
| 'extracting'
| 'synthesizing'
| 'done'
| 'error';