Field Notes · GEO
Is Your Site Blocking GPTBot?
A practical audit of AI crawler access. If answer engines cannot fetch your pages, they cannot cite you, and most sites block the wrong bots without ever knowing it.
Before an AI answer engine can cite your brand, it has to fetch your page. If a line in your robots.txt, a firewall rule, or a Cloudflare bot setting turns the fetcher away, you are invisible in the answer no matter how good your content is. This is the quietest way to lose AI visibility, because nothing breaks. Your site loads fine for humans, your rankings look normal, and the answer engines simply skip you.
The fix starts with one distinction that most teams get wrong: there are two kinds of AI crawlers, and blocking one is a reasonable business decision while blocking the other erases you from the answers. This is a practical audit you can run in an afternoon to find out which one you are doing.
What is the difference between a training crawler and a live-answer fetcher?
A training crawler collects pages to train future models, while a live-answer fetcher pulls your page in real time to build an answer a user is reading right now. They are different bots with different user-agents, and they serve opposite purposes. Blocking training crawlers keeps your content out of model training sets. Blocking live-answer fetchers keeps your content out of the answers users see today.
OpenAI splits these cleanly. GPTBot is the training crawler. OAI-SearchBot is the bot that fetches pages to surface in ChatGPT search results. Google does the same: Googlebot crawls for search, and Google-Extended is a separate control that governs whether your content is used for Gemini and AI training, without affecting how you rank. Treating all of these as one undifferentiated swarm is the root mistake.
- GPTBot (OpenAI): training crawler. Blocking it keeps you out of training data, not out of ChatGPT search.
- OAI-SearchBot (OpenAI): live search fetcher. Block this and you can disappear from ChatGPT's cited answers.
- ClaudeBot and Claude-User (Anthropic): crawling and user-initiated fetches. Block these and Claude has less to retrieve from you.
- PerplexityBot (Perplexity): crawls pages Perplexity may cite. Blocking it reduces your odds of being a Perplexity source.
- Google-Extended (Google): a training and AI-use control. It does not change your Google Search ranking.
How do you read robots.txt for the AI user-agents?
Open yourdomain.com/robots.txt in a browser and search for each AI user-agent by name, then check what each block allows or disallows. Robots.txt is a plain-text file at the root of your domain, and it lists rules grouped under User-agent lines. A rule of Disallow: / under a given agent tells that bot to stay off the entire site.
Watch for three patterns. First, an explicit block, where an AI agent is named with Disallow: /. Second, an inherited block, where User-agent: * has Disallow: / and an AI bot has no rule of its own, so it falls under the catch-all. Third, a path block, where a bot is allowed on the site but disallowed from the exact templates you care about, such as /blog/ or /products/. Many sites that think they are open are quietly blocking AI fetchers through the wildcard rule they wrote years ago.
- Search the file for: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended.
- Confirm whether each one is named directly or is inheriting the User-agent: * rule.
- Check the disallowed paths, not just the domain root, because a sitewide allow with a blocked /blog/ still hides your best content.
- Remember robots.txt is voluntary. Well-behaved bots obey it, but it is not a security boundary.
Can a firewall or CDN block AI crawlers even when robots.txt allows them?
Yes, and this is the block most teams never find, because it lives in your CDN or firewall rather than in any file you edit. A clean robots.txt means nothing if Cloudflare, a WAF, or a hosting-level bot filter returns a 403 or a challenge page to the AI fetcher. The request never reaches your content. From the bot's side, your page simply does not exist.
Cloudflare is a common culprit because it now ships bot controls that can block or challenge AI crawlers by default or by toggle, and many site owners enable them without realising live-answer fetchers are in scope. Generic bot-fight modes, rate limits, and managed rules can all catch legitimate AI fetchers. The symptom is specific: humans and Googlebot load the page, but a request sent with an AI user-agent gets a block page, a CAPTCHA, or a silent timeout.
- Cloudflare: check Bot Management, the AI crawler toggles, Bot Fight Mode, and any custom firewall rules referencing user-agents.
- WAF or hosting bot filters: confirm they are not challenging or rate-limiting AI agents at the edge.
- Server config: scan .htaccess or nginx rules for user-agent denies that an old plugin or security hardening step left behind.
- Symptom to look for: a 403, a JavaScript challenge, or a timeout that only appears for AI user-agents, never for a normal browser.
Why does JavaScript rendering hide content from AI crawlers?
Many AI fetchers read your raw HTML and do not run JavaScript, so any content that only appears after a script executes is invisible to them. If your page ships a near-empty HTML shell and builds the real text client-side, the crawler may fetch the page, get a 200 response, and still see almost nothing worth citing. The block here is not a rule. It is your rendering strategy.
This hits single-page apps and JavaScript-heavy frameworks hardest, along with content loaded by tabs, accordions, lazy-load, or infinite scroll. The test is direct: view the raw source of the page, not the rendered DOM in your inspector, and confirm your headings, body copy, and answers are actually present in the HTML that arrives. If the source is a hollow shell, server-side rendering or static pre-rendering for that template is the fix.
How do you self-test by fetching your pages as each AI bot?
Request your own pages with each AI user-agent and compare the response, which tells you in seconds whether you are blocked, served a challenge, or served the content. The cleanest way is a command-line fetch where you set the user-agent string to match the bot, then read the status code and the body. A 200 with your real HTML means you are reachable. A 403, a challenge page, or a redirect to a block screen means you are not.
Run the test for a few representative URLs, including your homepage, a key money page, and a deep blog post, because edge rules sometimes apply to specific paths. Compare each AI user-agent against a normal browser user-agent on the same URL. When the browser succeeds and the AI agent fails, you have isolated a bot-specific block to chase down in robots.txt, the CDN, or the firewall.
- Fetch each URL once per agent: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, and a standard browser as your control.
- Compare the HTTP status code: 200 is reachable, 403 or 401 is blocked, a challenge or redirect means a bot rule fired.
- Inspect the returned HTML for your headings and body text, to catch the JavaScript-rendering trap in the same pass.
- Cross-check the server logs to confirm real AI bots are visiting and what status you actually returned them.
What do you do when you find a block?
Decide per bot whether the block is intentional, then remove it only where you want to be cited. The decision is the same one we opened with: allow live-answer fetchers if you want to appear in AI answers, and allow or block training crawlers based on your own stance on model training. There is no single correct setting. There is only a setting that matches your goal, applied consistently across robots.txt, the CDN, and the server.
- Robots.txt: add explicit Allow rules for the live-answer fetchers you want, rather than leaving them to inherit a restrictive wildcard.
- Cloudflare or WAF: create an allow rule or exception for the AI user-agents you have chosen to permit, and confirm bot-fight modes are not overriding it.
- Rendering: move citable content into server-rendered or static HTML so it is present in view-source.
- Re-test after every change, fetching as each bot again, because edits to one layer can be undone by a rule in another.
The goal is a clear, deliberate posture: humans, search crawlers, and the live-answer fetchers you care about all get your real content, while anything you choose to exclude is excluded on purpose, not by an inherited rule nobody remembers writing.
An AI crawler-access audit is unglamorous and high-leverage: a one-line block can quietly cost you every AI citation you might have earned. Checking each layer, then fetching your own pages as each bot, is exactly the kind of work DarkHorse does before chasing harder GEO wins.
Questions people ask
Should I block GPTBot?
Blocking GPTBot only stops OpenAI from using your pages to train future models. It does not remove you from ChatGPT's live search answers, which use a separate bot called OAI-SearchBot. If you are comfortable with your content not feeding model training but still want to be cited in ChatGPT, block GPTBot and allow OAI-SearchBot.
Does blocking AI crawlers hurt my Google rankings?
No. Google-Extended controls whether your content is used for Gemini and AI training, and it is fully separate from Googlebot, which handles Search. You can block Google-Extended without affecting how you rank in normal Google results. The two systems use different user-agents and different controls.
How can I tell if Cloudflare is blocking AI crawlers?
Fetch one of your pages with an AI user-agent string and check the response. If you get a 403, a JavaScript challenge, or a managed-challenge page instead of your HTML, a bot rule is firing at the edge. Then check Cloudflare's Bot Management settings, AI crawler toggles, and any custom firewall rules that reference user-agents.
My robots.txt allows AI bots but I still am not cited. Why?
Robots.txt is only one layer. A clean file can be overridden by a CDN or firewall rule that blocks the fetcher before it reaches your content, or your page may rely on JavaScript to render text that the crawler never sees. Test by fetching the page as each bot and reading the raw HTML, then check view-source to confirm your content is actually present without scripts running.