Cloudflare’s new Bot Preference Sync automatically rewrites your robots.txt (the file that tells AI crawlers what they can and cannot access on your site) to match the bot policies you’ve already set in your dashboard. Available from the free tier to Enterprise and on by default for new customers, it closes a real and common gap between stated policy and actual enforcement. The tradeoff: a vendor now holds the default pen on a file that directly shapes your AI-search visibility.
Cloudflare announced the feature on August 21, 2026, one day after Search Engine Journal published an analysis documenting exactly the kind of policy drift the tool is designed to fix.
What Problem Does This Actually Solve?
Your robots.txt and your edge enforcement rules (the access controls running on the server that actually serves your pages) are two separate things that live in two separate places. The file is text you wrote once, possibly years ago. The enforcement rules are settings you’ve changed since. Nothing keeps them in sync with each other.
That gap matters in practice. Cloudflare’s announcement states that “when your stated preferences and your enforced rules disagree, some crawlers treat it as a basis to disregard your preferences or try to bypass your enforced rules.” The Search Engine Journal analysis cited a concrete example: its author’s own robots.txt had been welcoming Bytespider by name for months after they would have chosen otherwise, and they only caught it while writing about the issue.
Bot Preference Sync addresses this by auto-generating robots.txt entries from dashboard settings and inserting them between # BEGIN Cloudflare Bot Preference Sync and # END Cloudflare Bot Preference Sync markers, with existing content preserved underneath. Cloudflare says it can be turned on or off at any time.
How Are AI Crawlers Categorized?
Bot Preference Sync generates policy from three categories that Cloudflare introduced on July 1, 2026. Each maps to a distinct type of AI traffic:
- Search: Crawlers that index your content so they can answer questions about it later. Cloudflare’s documentation notes publishers should expect referral traffic or other equitable compensation in return.
- Agent: Automated activity acting in real time on a person’s behalf, such as chat fetch bots and browser-use agents (software that navigates the web autonomously to complete tasks).
- Training: Crawlers that take your content to train or fine-tune an AI model.
For each category, publishers can choose to block on all pages, block only on pages that display ads, or allow access entirely. Cloudflare’s BotBase (a searchable directory of every bot it tracks, available to Enterprise Bot Management customers) shows how each known bot is classified across these and other categories, including Transact, Data Collection, SEO, and Ads Verification.
For context on one major player: Google’s own documentation states that Google-Extended is the token publishers use to control whether their content may be used for training Gemini models. Google specifies explicitly that Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal, and that Googlebot always obeys robots.txt rules when crawling automatically.
What Changes for New Domains on September 15, 2026?
Starting September 15, 2026, new domains onboarding to Cloudflare receive updated defaults: bots classified as Training or Agent are blocked on pages that display ads, while Search remains allowed. Multi-purpose crawlers that combine Search and Training behaviors will also be affected by the new Training block defaults on that date. Cloudflare notes all customers can opt out at any time before September 15.
The defaults mean new site owners who never adjust their settings will, by design, block training and agent crawlers from their ad-monetized pages. Cloudflare’s blog illustrates why the right answer differs by business model: an e-commerce store may want everything crawled so its products surface when a shopper asks a chatbot for a recommendation, while a publisher monetizing pages with ads may want the opposite.
Where Does the Three-Category Model Fall Short?
Bot Preference Sync sets policy per category, not per individual crawler. The Search Engine Journal analysis flags this directly: the feature “cannot express what many websites actually do.” A publisher might want to allow one specific training crawler and block another, or permit a particular agent while restricting others. The current system does not expose that per-crawler granularity through the automated robots.txt output.
As of September 13, 2026, the feature had not yet appeared in Cloudflare’s bots changelog (which still ended at July 1), its bots documentation, or the SEJ author’s own robots.txt, despite being announced three weeks earlier. The product is real and deployed, but its public documentation trail was still catching up at time of that writing.
What This Means for AI-Search Visibility
Plain language first: your robots.txt is the first signal an AI crawler reads before deciding whether to index your content. If that file says one thing and your server does another, you’ve handed any crawler that wants to ignore you a documented reason to do so. Bot Preference Sync removes that inconsistency automatically. That’s genuinely useful.
The subtler issue is about who holds the default. When Bot Preference Sync is on by default, Cloudflare’s category definitions shape what your robots.txt says. A crawler that indexes content for AI-powered answers but also trains on that content could fall under the Search designation, depending on how Cloudflare classifies it. Whether that classification matches a publisher’s actual preference is a question worth checking before the feature runs.
For publishers trying to be present in AI-powered search results, the stakes are concrete. Blocking the Agent category by default, for instance, could reduce the chance that your content surfaces when an AI assistant fetches pages in real time on a user’s behalf. Allowing the right categories deliberately, rather than inheriting a vendor default, keeps that decision in the publisher’s hands.
Cloudflare’s Business Insights dashboard (for Enterprise Bot Management customers) adds meaningful data here: it shows how much value each crawler sends back to a business, making it possible to match policy to actual referral outcome rather than category assumption alone. That kind of visibility is what turns robots.txt from a static text file into a policy tool worth managing actively.
There is also a structural question the tool raises but cannot answer: if category-based blocking at scale becomes standard, will AI crawler operators be pressured to be more transparent about which category their bots actually belong to? A crawler that self-identifies as Search while also training on content would pass through a Training block unaffected. The robots.txt signal is only as useful as the classification system it relies on, and that classification still belongs to the platform, not the publisher.
Quick Checklist: Before Bot Preference Sync Runs on Your Domain
- Compare your current robots.txt against your actual Cloudflare bot enforcement settings to see whether a gap already exists.
- Decide which of the three categories (Search, Agent, Training) aligns with your business model before enabling the sync.
- Review how Cloudflare classifies the specific crawlers you care about most, using BotBase if you have Enterprise access.
- Check whether any crawlers you want to allow are currently blocked at the edge, or vice versa, before the file is rewritten.
- Determine whether the three-category model is granular enough for your needs, or whether you require per-crawler rules outside the automated robots.txt.
- After enabling, verify the generated markers actually appear in your live robots.txt file.
- Monitor Cloudflare’s bots changelog and documentation for updates, as both were lagging the announcement as of mid-September 2026.
What to watch: How AI crawler operators respond to category-based blocking at scale, and whether Cloudflare adds per-crawler granularity to Bot Preference Sync, will determine how useful the tool becomes for publishers with more specific access policies than three categories can express.
Sources: Search Engine Journal; Cloudflare Blog; Cloudflare Developers Changelog; Google Developers: Common Crawlers
Lam Nguyen · Hingewise
This article was drafted with AI assistance from the sources cited above and checked against them before publishing. The "What this means for AI-search visibility" section is Hingewise's editorial analysis, built on the frameworks and positions we use in our GEO work. How we work: Editorial policy.