AI crawlers / Claude-SearchBot
Claude-SearchBot
Anthropic · Claude search · Discovery / citation crawler
Anthropic's search crawler. It indexes content to improve Claude's search results, allow it to be eligible for citations in Claude search.
At a glance
Affects live AI answersYes
Affects search inclusion,
Used for model trainingNo
robots.txt tokenClaude-SearchBot
Source confidencedocumented · verified 2026-09-04
Allow it, this crawler decides whether you can be cited in this engine’s answers.
What Claude-SearchBot is
Claude-SearchBot navigates the web to improve search result quality for Claude users. Anthropic states it "analyzes online content specifically to enhance the relevance and accuracy of search responses."
This is the Anthropic agent that matters for being surfaced and cited in Claude's search, not ClaudeBot, which is for model training.
Anthropic publishes crawler IP ranges at claude.com/crawling/bots.json for verification.
Controlling Claude-SearchBot in robots.txt
Add the matching group to the robots.txt at your site root.
Allow Claude-SearchBot:
User-agent: Claude-SearchBot
Allow: /Block Claude-SearchBot:
User-agent: Claude-SearchBot
Disallow: /Check what your live site actually allows with the Robots.txt + AI-Bot Tester.
Source
Verified on 2026-09-04 against Anthropic, Does Anthropic crawl the web?. Confidence: documented, stated in the vendor’s own documentation.
The next question
Your page is AI-ready. But does AI actually cite you?
Being crawlable is step one. The AI Visibility Checker shows whether ChatGPT, Google AI Overviews, Perplexity and Gemini name and cite your brand, and who they name instead.
Keep going