When I think about how ChatGPT retrieves information, I find it fascinating that most sources it pulls in don’t make it to the final answers. According to a report by AirOps, a whopping 85% of the sources identified by ChatGPT never appear in its final response.
Why this matters to me. If I’m aiming to have my content mentioned in AI-generated answers, it’s clear that simply being discovered by the AI isn’t sufficient. Most pages that get retrieved ultimately don’t get the exposure I’m hoping for.
Key insight. It’s interesting to note that just because a page ranks and is retrieved doesn’t mean it gets cited. My content has to align closely with the prompt or the context it supports to be chosen.
Per the report: the focus shifts to how well I can optimize my content for selection in the AI synthesis process, beyond just showing up in the search results.
By the numbers:
82,108 citations appeared in final responses, but only 15% of the retrieved pages were mentioned. That means 85% of the pages that surfaced during research didn’t make it into the answers.
Citation rates also varied based on query type:
18.3% for product discovery queries, 16.9% for how-to queries, and 11.3% for validation searches.
Fan-out queries. I noticed that when ChatGPT generates an answer, it often triggers additional internal searches, resulting in a “second citation surface.” This stood out in the dataset findings:
89.6% of prompts prompted two or more follow-up searches. Fan-out searches expanded 15,000 prompts into 43,233 queries. Interestingly, 32.9% of the cited pages were results from these fan-outs and not the original prompt.
95% of fan-out queries had zero traditional search volume.
Google ranking correlation. I’ve learned that high rankings in Google significantly improve chances of citation:
55.8% of cited pages ranked within Google’s top 20. Pages in Position 1 were cited 3.5 times more often than those outside the top 20.
About the data. AirOps examined 548,534 pages from 15,000 prompts to understand how ChatGPT expands queries and selects which citations to include.
The study. For those interested in diving deeper, check out The Influence of Retrieval, Fan-out, and Google SERPs on ChatGPT Citations.
Inspired by this post on Search Engine Land.
Leave a Reply