mirror of
https://github.com/Memo-2023/mana-monorepo.git
synced 2026-05-16 15:39:40 +02:00
4a3295d1d0
NestJS-based web crawler service for structured content extraction. Features: - Depth-controlled crawling with URL pattern filtering - robots.txt compliance - HTML/PDF/Markdown content extraction - BullMQ job queue for async processing - Redis caching layer - Prometheus metrics
1 line
36 B
TypeScript
1 line
36 B
TypeScript
export const CRAWL_QUEUE = 'crawl';
|