till/managarten
1
0
Fork 0
mirror of https://github.com/Memo-2023/mana-monorepo.git synced 2026-05-15 05:41:09 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 4
managarten/services/mana-crawler/src/queue
History
Till-JS 4a3295d1d0 ✨ feat(mana-crawler): add web crawler service 
NestJS-based web crawler service for structured content extraction.

Features:
- Depth-controlled crawling with URL pattern filtering
- robots.txt compliance
- HTML/PDF/Markdown content extraction
- BullMQ job queue for async processing
- Redis caching layer
- Prometheus metrics
2026-01-29 22:00:36 +01:00
..
processors ✨ feat(mana-crawler): add web crawler service 2026-01-29 22:00:36 +01:00
constants.ts ✨ feat(mana-crawler): add web crawler service 2026-01-29 22:00:36 +01:00
processor.module.ts ✨ feat(mana-crawler): add web crawler service 2026-01-29 22:00:36 +01:00
queue.module.ts ✨ feat(mana-crawler): add web crawler service 2026-01-29 22:00:36 +01:00
queue.service.ts ✨ feat(mana-crawler): add web crawler service 2026-01-29 22:00:36 +01:00