Till-JS
|
4a3295d1d0
|
✨ feat(mana-crawler): add web crawler service
NestJS-based web crawler service for structured content extraction.
Features:
- Depth-controlled crawling with URL pattern filtering
- robots.txt compliance
- HTML/PDF/Markdown content extraction
- BullMQ job queue for async processing
- Redis caching layer
- Prometheus metrics
|
2026-01-29 22:00:36 +01:00 |
|