AI readiness is not a one-layer problem, and most brands are solving only the first. An audit of 50 major websites published by Search Engine Journal, with data captured on June 12, 2026, found that nearly all sites had made it easier for AI to access their content. Almost none had done what it takes for AI to genuinely understand it.
At the same time, Cloudflare launched its Content Signals Policy: a new extension to robots.txt (the file that tells bots which parts of a site they can access) that lets site owners declare, in machine-readable terms, what AI systems are permitted to do with content once they have fetched it. Together, both developments point in the same direction. The next frontier of AI search is not just about access. It is about understanding and control.
What Is the Three-Layer AI Readiness Framework?
The Search Engine Journal framework defines AI readiness across three distinct layers. Layer One asks: can AI fetch and parse your content without stumbling? Layer Two asks: can AI determine what your pages mean and who owns them? Layer Three asks: can AI agents interact with your site to complete tasks on a user’s behalf? Each layer builds on the one before it.
Across all three layers, the framework covers 27 audit elements: 11 in Layer One, 3 in Layer Two, and 13 in Layer Three. Protocols are classified as Established (production-ready standards already in active use), Emerging (real protocols gaining traction), or Frontier (standards still being debated). For scoring, 12 Established signals were assessed on a 0-to-2 scale. The highest score in the 50-site cohort went to Airbnb.com at 79.2%. On average, sites had implemented just over half of those Established signals, according to Search Engine Journal.
Why Can AI Fetch Your Content But Still Get It Wrong?
Nearly two-thirds of the 50 sites audited by Search Engine Journal leave the question of which AI bots can access which content “entirely to luck,” according to the report. That matters because not all AI bots serve the same purpose. Training crawlers (bots that ingest content to build AI models) and search or agent crawlers (bots that retrieve content to answer queries in real time) have meaningfully different implications for a site owner.
Layer Two is where the understanding problem is most acute. Without structured data signals like JSON-LD schema (a standardized format that labels page elements for machines: this is the price, this is the author, this is the brand) and entity mapping (a signal declaring what your brand is and what subject it is an authority on), AI systems have to guess. They may read a number on a product page and not know whether it is the standard price or a loyalty-member discount. They may read an article about “Jaguar” and not know whether it refers to the car brand or a football club. The Search Engine Journal report cites both examples as direct risks of under-investing in Layer Two attribution signals.
What Is Cloudflare’s Content Signals Policy?
Cloudflare’s Content Signals Policy adds a new section to robots.txt that expresses not just who can access content, but what they are allowed to do with it afterward. Three signals are defined: search (indexing for search results), ai-input (using content as input for AI-generated answers), and ai-train (using content to train AI models). Each signal accepts “yes,” “no,” or blank, with blank meaning no preference has been expressed for that specific signal. A site owner can permit search while expressing no view on training, for example.
Cloudflare’s rationale connects to traffic scale. The company projects that bot traffic will exceed human traffic on the internet by the end of 2029. By 2031, Cloudflare anticipates that bot activity alone will surpass the sum of all current internet traffic. In that environment, the existing binary choice for content creators, between keeping content open and risking unwanted use or locking it behind logins and losing audience reach, becomes increasingly untenable. The Content Signals Policy is Cloudflare’s proposed middle path.
What This Means for AI-Search Visibility
The simplest frame: most brands have unlocked the front door for AI, but skipped the welcome mat, the floor directory, and the service desk. Getting cited in an AI answer is harder than it looks. Getting cited accurately is harder still.
Here is a concrete example. Imagine a hotel booking site where the nightly rate, the member promotional rate, and the non-refundable advance-purchase rate all appear on the same product page in a similar visual format. A human reader uses context to understand which is which. An AI agent (a bot acting on a user’s behalf to find and book the best rate) that lacks structured disambiguation signals will either guess or surface the wrong number. That error does not appear as a broken page. It appears as a wrong AI-generated answer the brand may never see.
The Search Engine Journal report notes that stronger Layer Two signals do two things: they raise the likelihood of content being cited in relevant AI responses, and they reduce the risk of those responses misrepresenting a brand’s claims. Most sites audited appear to underinvest in Layer Two relative to Layer One.
On the Cloudflare side, the Content Signals Policy introduces a formally machine-readable way to separate three different uses of content that most robots.txt files currently treat as one. Whether major AI crawlers adopt this standard widely enough to make it operationally significant remains an open question. The policy is new, and its practical impact depends on voluntary compliance.
Hingewise’s read: the gap between retrievability (Layer One) and meaning (Layer Two) is the most commercially consequential gap in this audit, and also the least visible in standard analytics. A brand that earns AI citations but earns them inaccurately is, in some cases, worse off than one that earns no citations at all. An incorrect AI answer about pricing, policy, or product attributes circulates without a correction mechanism. That risk does not show up in traffic reports or ranking dashboards, which is precisely what makes it worth prioritizing now rather than after it surfaces as a customer complaint.
What to Check Before Assuming Your Site Is AI-Ready
- Check your robots.txt for AI user-agent (bot identifier) rules: does it distinguish between training crawlers and search or agent crawlers, or treat all bots identically?
- Audit your JSON-LD schema coverage: do key pages define prices, products, authors, and organization identity clearly at the structured-data level, not just in visible text?
- Look for entity signals: is there a clear, machine-readable declaration of who your brand is and what subject area it is an authority on?
- For precision-sensitive content such as pricing tiers, legal terms, or health information, check whether similar-looking numbers or claims are structurally distinguished, not just visually formatted differently.
- Review whether your current robots.txt expresses any preference regarding AI training use of your content, and whether that matches your actual preference.
- Map your current setup against the 12 Established signals in the Search Engine Journal framework, with extra attention to the three Layer Two attribution elements.
The layer worth watching longest is Layer Three: agentic readiness (the ability for AI agents to complete transactions or service interactions directly on a site). As AI agents become more capable of acting on users’ behalf, the Search Engine Journal framework already includes 13 elements tracking this layer. That infrastructure gap is still early, but it is the one most likely to translate directly into commercial outcomes as agent use grows.
Lam Nguyen · Hingewise
Sources: Search Engine Journal, June 2026; Cloudflare Blog, 2026
This article was drafted with AI assistance from the sources cited above and checked against them before publishing. The "What this means for AI-search visibility" section is Hingewise's editorial analysis, built on the frameworks and positions we use in our GEO work. How we work: Editorial policy.