All 14 tools / Crawlers, links and search

See which AI crawlers your robots.txt lets in or blocks, and get a draft llms.txt for the site.

AI crawlersFreeFor site owners and developers

Better than by hand

Applies the real robots.txt matching rules (longest user-agent group, then the wildcard), which are easy to misread by hand. We do not put a time on it.

Before: what you give it

The example input, as typed in. You give it a page address.

https://allthepossibles.com

After: what comes back

The first lines of a real run of the tool's own code on that input, fetched live when the sample was made, so the live page can differ from it. Nothing here is edited by hand.

https://allthepossibles.com: robots.txt found, llms.txt found
  allowed GPTBot                OpenAI, training
  allowed OAI-SearchBot         OpenAI, search
  allowed ChatGPT-User          OpenAI, on-demand fetch
  allowed ClaudeBot             Anthropic, training
  allowed Claude-Web            Anthropic, legacy
  allowed Claude-User           Anthropic, on-demand fetch
  allowed anthropic-ai          Anthropic, legacy
  allowed Google-Extended       Google, Gemini training
  allowed PerplexityBot         Perplexity
  allowed Perplexity-User       Perplexity, on-demand fetch
  allowed Bytespider            ByteDance
  allowed CCBot                 Common Crawl
  allowed Amazonbot             Amazon
  allowed Applebot-Extended     Apple, training
  allowed cohere-ai             Cohere
... full output on the tool page

What you get

Allowed, blocked or partial for nineteen named crawlers, whether llms.txt exists, and a drafted llms.txt

What it cannot do

Reads what robots.txt says; it cannot tell whether a crawler obeys it, and blocks at the firewall are invisible to it.

Who it is for

Site owners
Developers

Free and Pro

Free	10 runs a day per IP address.
Pro	No daily limit, 300 requests a minute.

Daily caps for every tool

Use it from your assistant

The tool is called check_ai_crawlers. Connect the server once, then ask for it by name or describe the job. Install steps.

claude mcp add --transport http toolbox https://toolbox.allthepossibles.com/mcp

curl -X POST https://toolbox.allthepossibles.com/api/ai-crawlers -H 'content-type: application/json' -d '{"url":"https://allthepossibles.com"}'

Try it here

The form starts with the same example input, so you can run it first and then swap in your own.

Site URL