Market Research & Insights
Wigolo Crawl
Local-first multi-page crawl with sitemap, BFS, DFS, and URL-map strategies, anchor-fragment dedup, rate limiting, robots.txt respect, and automatic local cache population. Use when the user wants to index documentation, crawl a docs site, extract all pages under a path, or says "crawl", "index this site", "get all the docs", "bulk extract". Prefer when crawled pages should land in a reusable local cache for later `cache` / `find_similar` queries.
- Publisher / source
- KnockOutEZ/wigolo (found on GitHub top repositories)
- Upstream record
- KnockOutEZ/wigolo
- Version
- Version 1
- Published
- Publication review
- Approved for publication
- Content fingerprint
a7f989971b33f38fdcf21b93f99e8ee8bad12639d5a0544fc9e61b51830fa1cd