Should you block GPTBot.
GPTBot collects pages to train OpenAI's models. Blocking it doesn't keep you out of ChatGPT's answers: a different crawler, OAI-SearchBot, does that job. The big AI companies now run separate crawlers for training and for answers, and you can decide on each one.
Training, search, and a person asking.
| Company | Crawler | What it does | If you block it |
|---|---|---|---|
| OpenAI | GPTBot | Collects pages to train OpenAI's models | Your pages are left out of training. ChatGPT search is not affected |
| OpenAI | OAI-SearchBot | Fetches pages for ChatGPT search | OpenAI says you are not shown in ChatGPT search answers |
| OpenAI | ChatGPT-User | Visits a page when a person asks ChatGPT to | OpenAI says robots.txt may not apply, since a person asked |
| Anthropic | ClaudeBot | Collects pages for training Claude | Your future pages are left out of training |
| Anthropic | Claude-SearchBot | Builds Claude's search results | Anthropic says you may be less visible in Claude's search |
| Anthropic | Claude-User | Fetches a page when a person asks Claude | Claude can't read your page for that person |
| Google-Extended | Controls Gemini training and grounding in the Gemini app | Gemini can't use your pages; Google says Search is not affected | |
| Perplexity | PerplexityBot | Builds Perplexity's search results; not used for training, Perplexity says | Your pages can drop out of Perplexity's answers |
From each company's own page, read 20 October 2026: OpenAI's page on its programs; Anthropic's page on its programs; Google's page on Google-Extended; Perplexity's page on its programs.
Block training, allow answers.
Many sites block the training crawlers and allow the search ones, so their pages stay out of model training and can still be cited. That is a policy choice, not a mistake, and the free AI Readiness Check reports training crawlers beside its score without counting a block against you.
Check your host too. On 1 July 2025 Cloudflare changed its default to block AI crawlers (Cloudflare), so a site can block them without anyone deciding to. The setting is in your Cloudflare dashboard.
4 steps, about 10 minutes.
Read your robots.txt
Open yoursite.com/robots.txt and look for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended and PerplexityBot.
Check your firewall and host
A security setting can turn crawlers away even when robots.txt allows them. Cloudflare's AI crawler setting is the most common one.
Run the AI Readiness Check
The free AI Readiness Check shows which AI crawlers can read one page of your site, in about 20 seconds.
Give it a day
OpenAI says ChatGPT search takes about 24 hours to act on a robots.txt change.
Asked before you ask.
Should I block GPTBot?
Block it if you don't want your pages used to train OpenAI's models. It won't keep you out of ChatGPT search, which uses OAI-SearchBot. If you want ChatGPT to recommend you, keep OAI-SearchBot allowed.
Does blocking GPTBot hurt my visibility in ChatGPT?
Not in ChatGPT search, according to OpenAI: GPTBot is for training, OAI-SearchBot is for search, and each setting is separate.
How do I block GPTBot but allow ChatGPT search?
In robots.txt, add the line User-agent: GPTBot followed by Disallow: /, and leave OAI-SearchBot allowed.
Do AI crawlers follow robots.txt?
The major AI companies say their crawlers do. OpenAI notes that ChatGPT-User, which visits a page because a person asked, may not follow it.
See what AI tells your buyers about you.
Plans from €79 a month before VAT, with all 7 assistants on every plan. Or start with the free AI Readiness Check.