Recently, I delved into an intriguing study exploring how enabling search impacts ChatGPT’s product recommendations. Remarkably, these changes affect a vast 80.2% of responses, as observed from an extensive analysis of 20,000 interactions conducted by Jeff Oxford, the founder and CEO of Visibility Labs.
In Oxford’s experiment, he executed 1,000 product-recommendation prompts, running each ten times with search enabled and ten times with it disabled.
Surprisingly, a mere 19.8% of products recommended without search were repeated in the results with search activated.
Search reshapes top suggestions. Even the products that ChatGPT frequently recommended without search seldom appeared once search was turned on. Among those consistently recommended in search-disabled responses, only 15.8% showed up when search was activated.
Oxford anticipated that highly recommended products would still dominate with search, but they turned out to have the least overlap.
Source mentions and visibility. This study also scrutinized whether products cited in ChatGPT’s sources appeared more frequently in recommendations, showing a modest correlation of 0.4 Pearson between source mentions and recommendation frequency.
Products mentioned more often in cited sources had higher Visibility Scores, based on the percentage of instances a product appeared for a given prompt.
The analysis didn’t prove that source mentions directly caused these recommendations.
Search refines the list. With search enabled, ChatGPT’s responses averaged 5.2 products compared to 6.2 without search.
On average, across ten runs for each prompt, there were 19 unique products returned with search enabled, versus 21.8 with it disabled.
Why it matters to us. These findings are crucial because they show how search significantly changes ChatGPT’s product recommendations, even for staple products. Also, products cited in sources may achieve greater visibility when search is enabled, though this study doesn’t conclusively show that source visibility is more influential than web visibility as a whole.
About the study. The analysis covered 1,000 product-recommendation prompts, with each run ten times with search enabled and ten times without. Product names were standardized for consistency. As an observational study, it didn’t establish a direct cause between source mentions and recommendation frequency.
The detailed report. For more insights, see the full study here.
Explore more. AI recommendation lists repeat less than 1% of the time: Study
Inspired by this post on Search Engine Land.
Leave a Reply