What are AI crawlers?
AI crawlers are bots that fetch web pages for AI companies. They fall into three groups, and each is named separately so a website can treat them differently:
- training crawlers collect pages to train future models: GPTBot for OpenAI, ClaudeBot for Anthropic and CCBot for Common Crawl, whose public archive is used in many AI training datasets
- search crawlers build the index an assistant searches when it answers: OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude and PerplexityBot for Perplexity
- user-triggered fetchers load a page because a person asked about it, such as ChatGPT-User, Claude-User and Perplexity-User
Google works differently. Googlebot crawls for Google Search, including AI Overviews and AI Mode. Google-Extended is not a separate crawler but a robots.txt setting that controls whether Google may use a site's content for training Gemini models and grounding Gemini's answers.
What happens if you block them?
It depends on which one. Blocking a training crawler means future models learn less from the site. It does not remove what earlier models have already learned, and it does not stop the site being found through search.
Blocking a search crawler has a more immediate cost. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as plain navigational links. Anthropic says blocking Claude-SearchBot can reduce a site's visibility in Claude's search results. For a business that wants to be recommended, these are the crawlers to keep open.
Google states that the Google-Extended setting does not affect a site's inclusion or ranking in Google Search. Blocking Googlebot itself, however, removes the site from Google Search entirely, AI Overviews included.
Who should block AI crawlers?
Businesses whose content is the product. A news publisher behind a paywall, a course provider, a research firm selling reports or a stock-photo library has a fair reason to stop its work being used to train models that compete with it, and may choose to block training crawlers while allowing search crawlers.
A hotel, travel agency, property developer or service business is in the opposite position. Its pages exist to be found and to describe the business accurately. For most of them, the bigger risk is an assistant that knows nothing about the business, or repeats an old directory's wrong phone number, because the official site was closed to it.
How can a site block AI crawlers without anyone deciding to?
More easily than most owners expect. A robots.txt copied from a template, a security plugin, a hosting provider's bot protection or a CDN setting can each block AI crawlers. CDNs now offer one-click AI-bot blocking, and some can also add their own rules to the top of the site's robots.txt.
It happened to this website. In November 2026 our own robots.txt explicitly allowed GPTBot, ClaudeBot and the other AI crawlers. At the hosting edge, however, a Cloudflare setting was refusing GPTBot, ClaudeBot, CCBot and Amazonbot with a 403 error, and a managed block added above our rules told GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and others to stay away. The search crawlers for ChatGPT, Claude and Perplexity were getting through; the training crawlers were not. Nobody had chosen that combination. On 14 November 2026 we set the training policy to allow and switched off the managed rules, so our own robots.txt is now the only policy the site serves.
How do you check your own site?
Three checks catch nearly every case:
- open the live robots.txt in a browser, at the site's address followed by /robots.txt, and read every rule, including any block added by the host or CDN
- request a page while identifying as each crawler, which a developer can do in a minute with a command-line tool, and check that the response is 200, not 403
- look at the CDN or hosting dashboard for bot, AI-crawler or firewall settings, and at the server logs or analytics for visits from the crawlers you expect
Repeat the checks after any change of host, CDN, theme or security plugin, because each can reset the defaults.
What did we decide, and why?
We allow all of them: search crawlers, user-triggered fetchers and training crawlers. CompanyFirst's pages exist to explain what the firm does, for whom and how, and we would rather AI systems learn that from our own pages than from guesswork. Our robots.txt still keeps form endpoints closed to every crawler, because they are not pages.
That is a decision each business should make deliberately, with the trade-off in view. The mistake is not choosing to block or allow; it is not knowing which one the site is actually doing.
Where this fits in the audit
Crawler access belongs to Layer 02 of the Visibility Playbook, crawl and index, and the account control behind it to Layer 00. The Digital Footprint Audit checks the live robots.txt, the responses each major search and AI crawler receives and the CDN and firewall settings, then records the business's chosen policy so the next change of host does not quietly undo it.