A controlled experiment posted to arXiv on September 14 finds that the citation advantage held by top-ranked sources in AI search results is far smaller than raw data suggests. When researchers swapped the same two competing sources while keeping everything else identical, the position effect on AI citations fell from 42.3 percentage points to effectively zero. Structured formatting did shift citation credit, but in a narrower way than many content guides assume.
How Did Researchers Design a Fair Test of AI Citations?
Most citation studies compare different pages ranked in different positions, making it impossible to separate content quality from position. This study tried to isolate each variable by replaying the same conversation with only one element changed at a time.
Researchers Sriram Selvam and Anneswa Ghosh had a GPT-5.4 agent (using Exa as its search provider, meaning Exa supplied the web results the agent worked with) answer 130 common questions by running its own web searches. Every message, query, and retrieved result was archived verbatim. From those transcripts, the team identified 113 document pairs where two different pages appeared in the same search call and both verifiably supported the same specific fact. A later blinded human review confirmed 103 of those pairs as genuine “same-claim competitions.”
The researchers then replayed each frozen conversation four ways: placing one page above or below the other, and presenting its text either as plain paragraphs or as a version rewritten with headings, lists, or a table. Only the final answer was regenerated. Both text versions were AI rewrites created primarily by Grok 4.3, with a separate Grok pass checking factual accuracy. The study was preregistered before results were collected, and all transcripts were hash-verified to confirm no other variable had changed.
Does Ranking Position Actually Cause More AI Citations?
The raw numbers look dramatic. But the controlled swap test tells a different story.
In the archived transcripts, documents appearing at rank 1 (first position in Exa’s results for a single search call) were cited 85.1% of the time. Documents at rank 5 were cited 42.8% of the time, according to the arXiv paper. That is a 42.3 percentage-point gap.
When the researchers swapped the same document to a higher position within its competing pair, the estimated citation lift was +7.9 percentage points in the main replay set. That result did not reach statistical significance after adjusting for multiple tests. In a held-out confirmation set of 56 pairs where only order was swapped, the measured effect was exactly 0.0 percentage points, with a 95% confidence interval from -5.4 to +5.4.
The paper explains the apparent contradiction directly: search engines place more relevant pages earlier, so the raw rank-1 vs rank-5 gap reflects both position and content quality. When the same document is moved against a matched competitor, the quality advantage disappears and the position effect becomes negligible.
This distinction matters for how practitioners interpret their own tracking data. A page consistently cited at the top of AI results is likely there because it is more relevant, not because position alone drives citations.
Does Structured Formatting Help Content Get Cited?
Formatting did show an effect, but not the one most optimization guides describe.
Pages rewritten with headings and lists received an average of 0.50 more citation markers per answer compared to the same pages written as plain paragraphs, with a 95% confidence interval from +0.20 to +0.84 (Holm-adjusted p=0.033), per the arXiv paper. The answers in these tests were heavily cited documents, with a median of 29 citation markers per answer across six source documents.
The total number of citations per answer did not increase, and the count on the competing page barely changed. The researchers describe this as citation credit being redistributed toward the structured page rather than the page earning more “entries” into the answer.
The pre-specified primary test, whether structured formatting increased the likelihood of the target page being cited at all, found a +4.5 percentage-point effect with a 95% confidence interval from -1.4 to +10.4 (p=0.168). The paper flags this as inconclusive. The study’s design could reliably detect effects of roughly 8.5 percentage points or larger, so a genuine effect smaller than that would have been invisible here.
One secondary finding from the arXiv paper adds context about how AI agents encounter formatted content in practice: among 1,750 archived result-text records in the study’s dataset, 93.2% already contained markdown-style headings, 76.6% contained list-marker lines, and 21.3% contained table-like rows. The baseline is already high.
Why Is One Citation Test Not Enough?
Even holding every input constant, AI citations are not fully deterministic.
The researchers reran 120 responses using identical inputs and found that the binary decision to cite or not cite the target page changed in 15% of cases, roughly one in seven, according to the arXiv paper. They estimate that approximately 45% of the variance in a single run’s measured effect is attributable to model randomness (the random variation introduced each time the model generates a new response) rather than to the formatting or position variable being tested.
Additionally, SparkToro reported that ChatGPT and Google’s AI Overviews each produced the same brand list less than 1% of the time when given the same prompt repeatedly. The researchers recommend rerunning citation tests multiple times and reporting consistency across runs, not just a single result.
The paper also notes a relevant finding from an Ahrefs report published in May: pages cited by AI were about three times more likely to include JSON-LD schema (structured data markup embedded in a page’s code that helps systems read its content), but adding schema did not clearly increase citations. That is a correlation without a confirmed causal direction.
What This Means for AI-Search Visibility
Think of it this way: if you see a competitor’s page getting cited constantly in AI answers, the most likely reason is that it covers the topic better, not that it happens to rank first. Chasing its position won’t close the gap if the content quality difference is the real driver.
This is Hingewise’s read of what the findings actually change in practice:
The 42-point raw position gap has been cited repeatedly in vendor reports and content audits as evidence that ranking higher in AI search is the dominant lever. This study’s controlled design separates that story into two parts. The quality difference between rank-1 and rank-5 pages is real and substantial. The position effect itself, when isolated, is small and possibly zero. This means strategies built around “getting into the top result slot” may be conflating quality improvement (which works) with pure rank manipulation (which this study cannot confirm).
The formatting finding is more actionable, but also narrower than it first appears. Structured content redistributes citation markers within an answer rather than earning a new “slot” in the answer. For a brand trying to increase its share of visible citations in a long AI response, that redistribution matters. It does not appear to be the difference between being cited and not being cited.
The noise finding is the one most practitioners will find uncomfortable. If 15% of citation decisions flip between identical runs, then a single-session brand tracking snapshot has a meaningful error rate baked in. Any attribution model that treats a citation as a stable “win” or “loss” is working with noisier data than it looks. Multi-run sampling is not a nice-to-have for citation research; it is a methodological baseline.
One implication the paper does not address directly: the study tested one agent (GPT-5.4), one search provider (Exa), and one category of content (everyday questions). How these effects translate to other AI systems, different query types, or pages with very different quality gaps between competing documents remains untested.
Before Drawing Conclusions from Your AI Citation Data: A Quick Checklist
- Run the same prompt at least 5-10 times before treating a citation outcome as reliable, given the 15% flip rate on identical inputs.
- Separate “pages retrieved” from “pages cited” in your tracking: a page not retrieved cannot be cited, but retrieval and citation are different decisions made at different stages.
- When you see a large citation gap between your page and a competitor’s, check content quality and relevance first before adjusting formatting or structure.
- If testing structured formatting, measure change in citation marker count per answer, not just whether the page was cited at all, as the two effects appear to differ in size and certainty.
- Note which AI system and which search provider your data comes from, since these results apply specifically to GPT-5.4 with Exa and may not generalize.
- Treat correlation data (schema, word count, formatting in observational studies) as hypothesis generators, not confirmed optimization levers, until a controlled test isolates the variable.
The CiteChoice framework introduced in this paper, which tests competing same-claim document pairs rather than comparing different pages at different ranks, represents a more rigorous standard for citation research. Whether it gets adopted more broadly by AI vendors or independent researchers is worth watching as the field develops methodological norms.
Source: CiteChoice preprint, arXiv, September 14; Search Engine Journal. The arXiv paper has not been peer-reviewed.
Lam Nguyen · Hingewise
This article was drafted with AI assistance from the sources cited above and checked against them before publishing. The "What this means for AI-search visibility" section is Hingewise's editorial analysis, built on the frameworks and positions we use in our GEO work. How we work: Editorial policy.