ChatGPT’s fetch bot, formally called ChatGPT-User, reached disallowed pages on nearly half of the European sites that had explicitly blocked it, according to TollBit’s State of the Bots report for H1 2026. OpenAI’s explanation is straightforward: because a real user triggered the request, robots.txt (the standard file webmasters use to tell automated bots where not to go) may not apply.
What does TollBit’s H1 2026 data actually show?
Across European sites in TollBit’s dataset, about 15% of identified AI page-fetchers reached URLs those sites had marked as disallowed. ChatGPT-User, Bytespider, and Youbot each accessed disallowed pages on nearly half of the European sites that had explicitly listed them. Among those three, ChatGPT-User reached the highest number of sites.
Newer agents look quite different. Only 9% of European websites disallow Claude-User, compared to 26% in North America. Perplexity-User sits at 13% in Europe versus 26% in North America. Most of the newest agents show disallow rates in the single digits across Europe. ChatGPT-User is the clear exception to that trend.
The core tension in TollBit’s report: ChatGPT-User is both the most-blocked AI fetch agent of its kind and the one that reached disallowed pages on the most sites. More sites tried to stop it than any comparable bot. It also got through more than any comparable bot.
Why does OpenAI say robots.txt may not apply?
OpenAI’s bot documentation on developers.openai.com explains that ChatGPT-User visits a page when a ChatGPT user explicitly asks a question requiring that page to be fetched. Because the action originates from a human user rather than an automated crawl, OpenAI argues robots.txt rules may not apply to those requests.
Perplexity holds a similar position: its Perplexity-User agent generally ignores the file for the same user-initiated reason, as reported by Search Engine Journal. Anthropic takes the opposite stance and states that all three of its bots respect robots.txt. TollBit’s methodology treats any request to a disallowed URL as a bypass, regardless of what the operator claims as justification.
Does blocking ChatGPT-User remove your site from ChatGPT search results?
Based on OpenAI’s own documentation, probably not. The bot responsible for whether a site appears in ChatGPT search results is OAI-SearchBot, not ChatGPT-User. These two bots serve different functions, and their robots.txt settings are independent of each other.
According to developers.openai.com, a site can allow OAI-SearchBot (to maintain ChatGPT search visibility) while disallowing GPTBot (OpenAI’s training crawler, which ingests content to improve the underlying model) at the same time. Those controls do not interact.
In practice, a site that blocks only ChatGPT-User to prevent AI traffic may have made a trade-off it did not intend: limiting one fetch pathway while leaving ChatGPT search indexing intact, and potentially giving up search visibility if both bots were blocked together. Server logs or CDN (content delivery network, the infrastructure layer that routes web traffic) records show what actually arrived. Robots.txt only records what was requested.
What is Cloudflare changing, and when?
Cloudflare announced an update to how it classifies and manages crawlers, moving enforcement to the network layer rather than relying on each bot to voluntarily comply. For bots Cloudflare recognizes, compliance is no longer left to the crawler itself.
Starting September 15, 2026, new domains added to Cloudflare will have Training and Agent crawlers blocked by default on pages carrying ads. Search crawlers will remain allowed by default.
Cloudflare’s updated classification groups bot behavior into three categories, according to its blog post:
- Search: bots that proactively collect and index content to answer future queries. Site owners should expect referral traffic or equivalent value in return.
- Agent: automated behavior acting in real time on a user’s behalf, including chat fetch bots like ChatGPT-User and browser-use agents like Gemini or Claude driving a live browser session.
- Training: crawlers that absorb content to train or fine-tune an AI model, permanently incorporating it into the model’s architecture.
Cloudflare’s post notes that many crawlers fall into more than one category simultaneously, and that separating crawlers by purpose creates more transparency for site owners about why a given bot is visiting.
What this means for AI-search visibility
Many sites blocking “AI bots” are targeting the wrong bot for the outcome they actually want. Here is a simple way to think about it: if you put a lock on your side door, but the front door is what determines whether you show up on the map, locking the side door does not take you off the map. It just controls one entrance, and even that control now has an asterisk attached.
This is Hingewise’s read of what the data reveals:
First, robots.txt was designed for a world with one kind of automated visitor: a crawler building a search index. That world no longer exists. A single AI assistant now sends at least three distinct bot types, each with a different purpose and a different relationship with robots.txt. Treating them as one category produces the wrong defensive posture.
Second, the user-initiated loophole is structurally significant, not just a technicality. If every major assistant now fetches pages on behalf of real users in real time (and they increasingly do), then robots.txt as a gating mechanism for those fetch requests becomes largely advisory. Cloudflare’s move to enforce at the network layer is a direct acknowledgment of that gap: when compliance depends on the bot’s own good behavior, it is not enforceable control.
Third, and this is the point most coverage skips: the ChatGPT-User and OAI-SearchBot distinction changes the calculus for brands trying to manage AI visibility. A site trying to appear in ChatGPT’s answers needs OAI-SearchBot to index it. Blocking ChatGPT-User alone does not affect that. A site trying to keep content out of live ChatGPT responses faces a harder problem, because the user-initiated argument means that even a correctly configured disallow may not hold.
Hingewise’s view: the more useful question is not whether to block these bots, but which bot controls which outcome for your visibility. Getting that mapping wrong means spending effort on controls that do not match the goal. Cloudflare’s three-category framework (Search / Agent / Training) is the clearest public taxonomy available for making that mapping as of mid-2026.
What to check before changing your robots.txt or bot settings
- Identify which bots you currently disallow and which category each falls into: Search, Agent, or Training.
- Check your server logs or CDN records to see which agents actually visited, including any that arrived despite being listed as disallowed.
- Confirm whether your robots.txt targets OAI-SearchBot (controls ChatGPT search visibility) separately from ChatGPT-User (controls real-time fetch requests), since OpenAI treats them as independent settings.
- If you use Cloudflare, review the new default settings effective September 15, 2026 for new domains, specifically the automatic block on Agent and Training crawlers for pages with ads.
- Check each AI company’s stated policy before assuming uniform behavior: Anthropic states all three of its bots respect robots.txt; OpenAI and Perplexity both argue user-initiated requests fall outside its scope.
- Match each disallow rule to a specific outcome: blocking training crawlers, limiting search indexing, and limiting real-time fetch are three separate goals that may require separate bot entries to address.
The user-initiated loophole argument from OpenAI and Perplexity has not been tested in a formal regulatory or legal setting, as of TollBit’s H1 2026 report. Whether that argument holds up under closer scrutiny from regulators or publishers is the question most worth tracking in the second half of 2026.
Sources: TollBit, State of the Bots report H1 2026, via Search Engine Journal; OpenAI, Bot documentation, developers.openai.com; Cloudflare, Content Independence Day: AI Options, blog.cloudflare.com, 2026.
Lam Nguyen · Hingewise