For sites that want their content picked up by AI systems, sitemap naming and RSS setup now matter beyond Google. Google’s John Mueller said on the Search Off the Record podcast (Oct. 1 episode, titled “Do sitemaps still matter?”) that he has seen AI crawlers access his sitemap and RSS files in his own server logs.
Why can’t you submit a sitemap directly to AI crawlers?
Unlike Google Search Console, AI training systems generally offer no submission interface. Mueller said on the Oct. 1 episode that AI training crawlers “usually don’t have any kind of a Console or any setup where you can submit a sitemap file.” That makes passive discoverability, through predictable file names and robots.txt, the primary lever site owners have.
For sites that want AI crawlers to find their content, Mueller recommended two approaches on the podcast:
- Use the default file name: sitemap.xml at your site root.
- Maintain an RSS feed. Feeds are usually linked in a page’s HTML
<head>, making them easy for any crawler to discover without a direct URL tip-off.
A robots.txt file can also carry a Sitemap directive (a line that points crawlers to the sitemap file’s location). As Search Engine Journal notes, the sitemaps protocol says this line is independent of the user-agent line, so it is not tied to any one crawler’s rules.
What did Mueller actually see in his server logs?
Mueller said he has personally seen AI crawlers access both his sitemap file and his RSS files in his server logs. He did not name specific crawlers, and said he does not know whether AI companies document this behavior or what they do with the files after retrieving them.
For site owners who want to keep a sitemap private, Mueller suggested giving it an unusual file name, omitting it from robots.txt, and submitting it directly to Google via Search Console. The tradeoff: other systems, including Bing and AI crawlers, will not find it. Bing would likely require its own separate submission.
Is llms.txt a real replacement for sitemap.xml?
Not currently. When asked whether llms.txt (a Markdown file format some sites publish to describe their content to AI systems) could replace an XML sitemap, Mueller said Google’s systems cannot use it as a sitemap because it lacks the strict format XML requires. “I think the hope is bigger than the reality,” he said on the podcast. “Currently none of this happens.” He is fine with sites experimenting with it, but said he wouldn’t rely on it.
Mueller had said in August that the only crawlers on his test sites that claimed to accept Markdown files were SEO tools. In May, Google’s AI optimization guide listed llms.txt among the tactics sites don’t need for generative AI features, according to Search Engine Journal’s earlier coverage.
Why does Search Console show “Couldn’t fetch” for a valid sitemap?
Mueller described two causes on the Oct. 1 podcast that have nothing to do with whether the sitemap file itself is technically correct. First, host load: Google’s systems may simply be too busy when they attempt the fetch, and Search Console logs it as “Couldn’t fetch.” Second, crawl demand: Google may skip the sitemap when its systems see no need to crawl much more from a site.
“The crawl demand is very often based on the perceived quality of a website,” Mueller said, adding that “it’s not purely a technical thing.” In February, he told a Reddit user that Google will not use a sitemap if it isn’t convinced there’s new and important content to index. Google’s Search Console Sitemaps report help page also lists low crawl demand among the reasons a sitemap cannot be fetched, alongside robots.txt blocks, unresolved manual actions, and wrong URLs.
What this means for AI-search visibility
The practical gap here is straightforward: with Google, you submit a sitemap and get a confirmation. With AI crawlers, there’s no inbox to submit to. Your sitemap either gets found on its own, or it doesn’t.
Picture a site with a custom sitemap named content-index-v2.xml, not listed in robots.txt. Google can still find it because it was submitted to Search Console. An AI crawler running on its own schedule has no way to know it exists. The default name and the robots.txt Sitemap directive are the main passive signals available to those crawlers.
RSS feeds are underappreciated in this context. They are already structured and machine-readable, they are typically linked from a page’s HTML head, and Mueller’s own logs show AI crawlers accessing them. For a site publishing content regularly, a well-maintained RSS feed is, in Hingewise’s view, one of the more practical and low-effort signals to keep current for AI discovery.
On llms.txt: in Hingewise’s view, there is little public evidence that major AI search or training crawlers rely on the format in a way site owners can confirm. Prioritizing llms.txt setup over maintaining accurate XML sitemaps and RSS feeds is likely a misallocation of effort right now.
The crawl-demand point has an implication worth flagging: low-quality sites face a compounding problem. If Google reduces crawl demand because it sees weak content signals, AI crawlers that use similar quality indicators may also find less on those sites. The sitemap file may be discovered, but what it points to still has to be worth indexing. Good file naming gets the crawler in; content quality determines what happens next.
Mueller’s most actionable suggestion was also his lowest-tech one: check your server logs. That’s where he saw AI crawlers. For site owners without access to Google’s internal tooling, server logs remain the most direct way to confirm whether AI crawlers are reaching your sitemap or RSS files at all.
Quick checks before your next sitemap review
- Confirm your main sitemap uses a predictable default name, such as sitemap.xml, at your site root.
- Check that your robots.txt file includes a Sitemap directive pointing to it.
- Verify your RSS feed URL is included in the HTML
<head>of your pages. - Pull server logs and filter for known AI crawler user-agent strings to see which bots are accessing your sitemap or feed URLs.
- If Search Console shows “Couldn’t fetch,” investigate crawl demand (content quality signals) before assuming a technical error in the sitemap file itself.
- If you’re testing llms.txt, treat it as experimental and don’t let it displace your XML sitemap maintenance.
The most concrete development worth monitoring: whether major AI platforms eventually introduce a sitemap submission console of their own. Mueller’s comments suggest that most AI training crawlers offer no such mechanism yet, and the discoverability gap between Google and AI crawlers is still entirely on site owners to bridge through passive signals.
Source: Search Engine Journal, Oct. 5 | Google Search Console Help: Sitemaps report
By Lam Nguyen, Hingewise
This article was drafted with AI assistance from the sources cited above and checked against them before publishing. The "What this means for AI-search visibility" section is Hingewise's editorial analysis, built on the frameworks and positions we use in our GEO work. How we work: Editorial policy.