The three-point gain in your AI citation share this week might mean nothing. New research published in April 2025 shows that a single reading from any AI visibility tracking tool is often too noisy to separate a real shift from random fluctuation, and the amount of data required to get a trustworthy number varies widely by platform and topic.
Why Does Your AI Visibility Score Keep Changing?
Because the models behind tools like ChatGPT, Gemini, and Perplexity are built with deliberate randomness. Ask the same question twice, and the system may surface different sources each time. Citation shares (the percentage of answers where your brand or content appears) are therefore snapshots of a moving target, not stable measurements.
A prior paper by Ron Sielinski, co-founder of IQRush, illustrated this with a concrete example. When testing SearchGPT on running gear queries, Tom’s Guide appeared in roughly 9.5% of citations, while Runner’s World accounted for about 6.0%, according to data published in that earlier paper. On first look, Tom’s Guide appears to be the clear winner. But the margin of error (the statistical range within which the true figure likely falls) on both numbers was wide enough that they overlapped. A 3.5-point difference at those sample sizes does not confirm who is actually ahead.
How Much Data Do You Need Before a Ranking Is Reliable?
The new IQRush preprint (a paper not yet peer-reviewed), released April 11, 2025, addresses this directly. It defines two conditions that must both be true at the same time for a ranking to be considered trustworthy.
- The order must stabilize. Early on, adding new answers reshuffles rankings because no site has established a clear lead. Once enough answers accumulate, the leading sites separate and the order stops shifting.
- The gap between top-ranked sites must exceed the margin of error. If two competitors are nearly tied, even a stable-looking order may reflect random variation rather than genuine performance.
Across 30 platform-topic tests covering three engines, the IQRush paper found that meeting both conditions required between 33 and 94 answers containing citations. Three of the 30 tests never reached this point even after 125 questions, all on SearchGPT, where top sites remained too similar to distinguish cleanly.
A separate, unaffiliated team of researchers at the University of St. Gallen (Julius Schulte, Malte Bleeker, and Philipp Kaufmann) published their own study in April 2025 and reached the same conclusion through an independent dataset: a single reading is unreliable, and repeated sampling is required before results can be trusted.
Does the Platform You Track Change How Much Data You Need?
Yes, and not in the direction most would expect. According to the IQRush paper, the key variable is not how many citations an answer contains, but how much independent information each answer actually carries.
Gemini tends to concentrate citations on a small set of sites within a single answer, so many of those citations carry overlapping information. SearchGPT returns fewer citations per answer but distributes them more broadly, meaning each answer carries more independent signal than the raw citation count suggests. The same number of collected answers does not buy equal confidence across platforms. A data budget that produces reliable rankings on Gemini may still leave you guessing on SearchGPT.
How Stable Are Top-Ranked Sites Compared to the Middle of the List?
The top positions are the most defensible. With enough answers, leading sites pull away from the middle of the pack. But even the top 10 are not clean: the IQRush paper reports a typical margin of error of about five positions for a top-10 site, with one in five carrying an error wider than 10 positions. Rankings in the middle and bottom of a list are closer to a coin flip between neighboring positions.
Separately, SparkToro research cited in the Search Engine Journal roundup found that AI tools return a different list of recommended brands more than 99% of the time when you ask the same question, a finding Rand Fishkin first noted in January 2025. That study left open the question of how many queries are needed before results stabilize. The IQRush paper is, in Fishkin’s framing, the clearest answer yet.
What This Means for AI-Search Visibility
The simplest way to think about this: if you ran a poll with only one respondent, you would not call it a survey. Most AI visibility dashboards today hand you a single-respondent number dressed up as a trend line.
Consider a concrete scenario. A brand manager refreshes the copy on a product page and checks their AI citation tool the next day. The score went up three points. That looks like a win. But the IQRush data suggests a three-point swing on SearchGPT can occur between runs with no content change at all, falling within the natural variability of successive runs. Without multiple readings before and after the change, that “win” cannot be separated from background noise.
This has an implication the research does not spell out but that follows directly from it: teams making content decisions based on single-run citation scores may be reacting to noise rather than to signal. The practical question to put to any AI visibility vendor is not what the number is, but whether they report a confidence range alongside it. A clean, precise-looking figure can actually be a warning sign. It may mean the vendor ran one collection pass and formatted the output, rather than sampling enough to know whether the number is stable.
There is also a subtler issue for brands tracking multiple competitors. The IQRush findings suggest that mid-list gaps are often within margin of error, meaning a brand ranked fifth and a brand ranked eighth may be statistically interchangeable. Decisions about which competitor to respond to in content strategy should account for whether the observed gap is real or within noise range.
One thing the research does not yet answer: how many runs to plan before starting a measurement campaign. The IQRush paper explicitly leaves a pre-test budget calculation for future work, noting that the required sample depends on each platform’s citation distribution pattern. A universal budget is unlikely.
Hingewise’s assessment: until vendors report ranges instead of point estimates, a single AI citation number is a hypothesis to test, not a result to act on.
Before You Act on Your Next AI Citation Report: A Quick Checklist
- Ask your vendor how many queries were sampled to produce the reported figure, and whether a margin of error is shown alongside it.
- Before concluding that a competitor outranks you, check whether the gap exceeds the margin of error on both figures. If it does not, the difference may not be real.
- Before crediting a content update for a citation gain, collect multiple readings before and after the change. A single before-and-after snapshot cannot separate your effort from ordinary variation.
- Note which platform you are measuring. A sample adequate for Gemini may be insufficient for SearchGPT, and vice versa, according to the IQRush paper.
- Treat top-of-list positions as more reliable than mid-list positions. Below the leading sites, neighboring positions may be within each other’s margin of error.
- If your vendor has no mechanism for reporting “not enough data yet,” that is worth flagging. Three of 30 tests in the IQRush study never produced a stable ranking.
What to watch going forward: whether AI visibility vendors add confidence intervals to their reporting, and whether Search Console begins attributing clicks to AI-generated results, which it still does not as of mid-2025. Until one or both of those happen, the verification work sits with the team running the measurement.
Sources: Search Engine Journal (July 2025); IQRush preprint by Ron Sielinski (April 11, 2025); University of St. Gallen study by Julius Schulte, Malte Bleeker, and Philipp Kaufmann (April 2025); SparkToro / Rand Fishkin (January 2025).
Lam Nguyen · Hingewise
Sources
This article was drafted with AI assistance from the sources cited above and checked against them before publishing. The "What this means for AI-search visibility" section is Hingewise's editorial analysis, built on the frameworks and positions we use in our GEO work. How we work: Editorial policy.