Marketing & Advertising
Llm Crawler Access Check
Check whether a website's robots.txt allows the AI crawlers that decide visibility in ChatGPT Search, Perplexity, Claude, Gemini, and Microsoft Copilot. Use when someone asks whether AI bots are blocked, whether to allow or block GPTBot, why a site never appears in AI answers, or wants a robots.txt review for AI crawlers. Reads only robots.txt, then returns a per-agent allow/block table, the exact rule responsible for each verdict, and the precise lines to change. Distinguishes training crawlers from the search crawlers that actually control citations.
- Publisher / source
- davepoon/buildwithclaude (found on GitHub top repositories)
- Upstream record
- davepoon/buildwithclaude
- Version
- Version 1
- Published
- Publication review
- Approved for publication · recorded passed scan 2026-11-05
- Content fingerprint
1e7a2d969bebc3c97d2bdb36bfd8f24c3e386f75f68411852e29f23ee94e3c55