till/managarten
1
0
Fork 0
mirror of https://github.com/Memo-2023/mana-monorepo.git synced 2026-05-24 03:16:44 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

feat(articles): server-side bulk-import worker (Phase 2)

Browse source 
apps/api/src/modules/articles/:
  - import-projection.ts: sync_changes → live LWW projection of jobs
    + items. Cross-user scan for claimable jobs, per-job item scan.
  - import-extractor.ts: per-item state-machine. Claim → fetch → write
    pickup + flip extracted, OR retry up to 3x then 'error'. All writes
    attributed to system:articles-import-worker actor (built inline so
    no shared-ai SystemSource extension needed for now).
  - import-worker.ts: 2s tick, pg_try_advisory_xact_lock keyed on 'ARTI'
    so multi-instance apps/api never double-processes. Concurrency 3
    pending items per job per tick. Job-counter rollups + status flips
    derived from current item states.
  - apps/api/src/index.ts: start the worker at boot.

Pipeline (server side):
  Client write articleImportItems(state='pending')
    → sync push → mana_sync.sync_changes
    → server-worker tick projects 'pending' items
    → extractFromUrl (shared-rss / Readability)
    → write articleExtractPickup row + flip item → 'extracted'

Phase 3 (client-side pickup consumer) and Phase 4+ (store + UI) follow.

Plan: docs/plans/articles-bulk-import.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Till JS 2026-04-28 21:33:49 +02:00
parent fc49198992
commit 5535f2da48
4 changed files with 723 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

7
apps/api/src/index.ts
View file

@ -41,6 +41,7 @@ import { moodlitRoutes } from './modules/moodlit/routes';
import { newsRoutes } from './modules/news/routes';
import { newsResearchRoutes } from './modules/news-research/routes';
import { articlesRoutes } from './modules/articles/routes';
import { startArticleImportWorker } from './modules/articles/import-worker';
import { tracesRoutes } from './modules/traces/routes';
import { writingRoutes } from './modules/writing/routes';
import { comicRoutes } from './modules/comic/routes';
@ -142,6 +143,12 @@ app.route('/api/v1/who', whoRoutes);
app.route('/api/v1/writing', writingRoutes);
app.route('/api/v1/comic', comicRoutes);
// ─── Background Workers ─────────────────────────────────────
// Articles bulk-import: ticks every 2s, advisory-lock-gated so multiple
// apps/api replicas never double-process. See
// docs/plans/articles-bulk-import.md.
startArticleImportWorker();
// ─── Server Info ────────────────────────────────────────────
console.log(`mana-api starting on port ${PORT}...`);