till/managarten
1
0
Fork 0
mirror of https://github.com/Memo-2023/mana-monorepo.git synced 2026-05-25 18:34:38 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

fix(articles, mana-ai): rollout-block hardening for sync_changes projections

Browse source 
Four cross-cutting fixes that make the bulk-import worker safe to run
under real production load. All four were called out as live-rollout
risks in the post-ship review of docs/plans/articles-bulk-import.md.

#1 — Same fieldMetaTime bug fixed in mana-ai
   The articles fix in 054b9e5be hoists the helper to its own file
   `apps/api/src/modules/articles/field-meta.ts`. The same naive
   `rowFM[k] >= localTime` LWW comparison existed in three more
   projections under services/mana-ai (missions-projection,
   snapshot-refresh, agents-projection). Once any F3 stamp lands
   beside a legacy-string stamp, the comparison evaluates
   `'[object Object]' >= 'ISO-…'` (false) and the older value wins.
   New `services/mana-ai/src/db/field-meta.ts` — same helper,
   deliberately duplicated (each service treats sync_changes as a
   read-only event log; sharing infra across services is out of
   scope here). All 61 mana-ai bun tests still pass.

#2 — Stale 'extracting' items recycle
   If the worker dies mid-fetch (OOM, pod restart), items stay in
   state='extracting' forever and the job never completes. New sweep
   at the start of `processOneJob`: items whose lastAttemptAt is
   older than 5 minutes get bounced back to 'pending' so the next
   tick re-claims them. STALE_EXTRACTING_MS tuned for the 15s
   shared-rss fetch + JSDOM-parse worst case.

#3 — Pickup-row GC
   Every 30 ticks (~once per minute) the worker hard-deletes
   articleExtractPickup rows older than 24h. Without this a stuck
   pickup-consumer (all tabs closed, Web-Lock mismatch) would let
   sync_changes accumulate without bound. Logs the row count when
   non-zero so we can spot stuck consumers in the wild.

#4 — DRY consent-wall heuristic
   Identical CONSENT_KEYWORDS + threshold lived in routes.ts AND
   import-extractor.ts. Hoisted to
   `apps/api/src/modules/articles/consent-wall.ts`; both call sites
   now share one heuristic.

Plan: docs/plans/articles-bulk-import.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Till JS 2026-04-29 00:53:39 +02:00
parent e99fea1938
commit b297f68ee4
10 changed files with 223 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

22
apps/api/src/modules/articles/routes.ts
View file

@ -26,30 +26,10 @@
import { Hono } from 'hono';
import { extractFromUrl, extractFromHtml } from '@mana/shared-rss';
import { looksLikeConsentWall } from './consent-wall';
const routes = new Hono();
const CONSENT_KEYWORDS = [
'cookies zustimmen',
'cookie consent',
'zustimmung',
'accept all cookies',
'consent to the use',
'enable javascript',
'javascript is disabled',
'please enable',
'privacy center',
'datenschutz­einstellungen',
'datenschutzeinstellungen',
];
const CONSENT_WORDCOUNT_THRESHOLD = 300;
function looksLikeConsentWall(content: string, wordCount: number): boolean {
if (wordCount >= CONSENT_WORDCOUNT_THRESHOLD) return false;
const haystack = content.toLowerCase();
return CONSENT_KEYWORDS.some((needle) => haystack.includes(needle));
}
function isValidHttpUrl(url: string): boolean {
try {
const u = new URL(url);