Whenever I type a question into an AI engine, I’ve noticed that the engine doesn’t just search for the exact words I typed. Instead, it explores a broader spectrum of possibilities. This behavior intrigues me.
Recently, I came across a fascinating study by Profound. They monitored 10,000 prompts across various AI platforms like ChatGPT, Copilot, and Perplexity over two weeks. The findings highlighted remarkable differences in how these AI engines search and process queries.
After tracking an incredible 2 million ChatGPT prompts, I found a surprising trend: shopping appears in less than 10% of them. Diving deeply into the data over nine months, it was clear that a staggering 79% of prompts simply never activated a shopping response.
What intrigued me further was the persistence of those that did trigger shopping. There was an impressive 83% chance they would do so again the following day. However, this persistence isn’t indefinite. Model updates seem to wash away those triggers overnight.
In my quest to understand these patterns, I analyzed 26 million prompts across 13,000 categories. The goal was to pinpoint where shopping emerges, how reliable this occurrence is, and what insights this holds for brands shaping their strategies on a platform where responses are sparsely shopping-oriented.
I’ve been asked numerous times about how to track prompts effectively, especially by those using tools like Profound, Athena, and Peec. The big question on everyone’s mind is, “Which prompts are worth tracking?” In this ever-evolving landscape, it’s challenging to determine what buyers are querying about my company when they use LLMs.
Currently, there isn’t a reliable data source that puts my mind at ease. Unlike traditional search with publicly available Keyword Planner data, it’s unlikely that OpenAI or Google will fully release this kind of data for analysis. Though there have been recent proposals by the UK CMA about Google and data transparency, I’m not holding my breath for significant change.
Long story short, LLM tracking feels like navigating a black box. So, are there any alternative data sources we can use to track which prompts? Perhaps.
Back in November, Jason Packer published an interesting report highlighting how ChatGPT searches accidentally leaked into Google Search Console reports, featuring PII. When this was confirmed by Ars Technica, OpenAI stated the problem affected only a small number of queries.
This confirmed, for me, that ChatGPT queries do appear in some Search Console profiles. While privacy implications are significant and beyond this article’s scope, it shows that LLM queries are not impossible to capture.
Additionally, Barry Schwartz has reported that AI Mode data is available in Search Console. This supports the idea that Search Console can track how users interact with LLMs.
Based on my analysis, it seems that AI data appears to come from this area. By applying specific filters, I’ve noted steady increases in impressions over recent months, coinciding with Google’s roll-out of AI Mode features.
So, how can I access user prompt data in Search Console? The key is focusing on longer queries. Using regex, we can filter queries with 10 or more words, unveiling prompt-like behavior:
1. Navigate to Search Console Performance > Search Queries
2. Select Add Filter > Query
3. Choose Custom Regex
4. Input: ^(?:S+s+){9,}S+$
This method revealed understandable, prompt-styled queries when applied to various properties. Though the actual data cannot be shared, examples such as “Map out a full day in Glacier National Park…” highlight the trend.
Mind you, there’s no direct evidence these queries originate from ChatGPT or similar AI platforms. It’s possible they reflect new user behavior patterns within Google.
Regardless, analyzing these conversational query patterns provides invaluable insight into how customers search using longer strings.
Will Critchlow wisely said, “we’re doing business, not science.” In our shift toward less attributed, zero-click data collection, the choice to leverage this available data is up to us.
Currently, my preferred tool for prompt analysis is Claude. Its results are reliably robust, and its visualizations are effective. Integrating Claude into existing frameworks streamlines the process.
After export, uploading prompt lists to Claude lets it perform behavioral analysis, identifying data themes and trends for better prompt tracking.
Posing specific questions to Claude about customer behavior opens a treasure trove of insights. Analyzing this data reveals learning opportunities I would not have anticipated.
For instance, I discovered searches probing a PR issue from over three years ago are still frequent and that searches often use one company as a benchmark against its competitors.
Finally, leveraging Claude to suggest new prompt-tracking methods, based on this data, offers an informed way to continually hone tracking efforts.
While there’s no definitive system for selecting which prompts to track, incorporating Search Console data provides a clearer direction. The insights derived can help unearth unique user prompts and discern scalable themes for ongoing data tracking.
