The past year has been a whirlwind as we all tried to grasp how to report on AI visibility and understand what it truly takes to be seen and cited by AI models.
Rand Fishkin’s recent study on the variability of AI responses pointed out how LLM outputs differ significantly from the stable and predictable nature of search rankings, making this KPI a challenging aspect of the analytics landscape.
The research illustrates a less than 1% chance that ChatGPT or Google AI will provide the same brand list in two different responses. They scrutinized thousands of prompts across various LLMs, revealing their unpredictable nature.
This unpredictability has led some in the SEO community to question the value of rank tracking on a broad scale. Despite these challenges, rank tracking remains a valuable, albeit misapplied, tool.
While AI response tracking is currently an unstable KPI, it proves to be incredibly potent when used as an analytical tool to inform content strategy.
I’m diving into why we should continue investing in prompt tracking and how this effort can illuminate our content strategy.
Why AI Visibility Tracking is Currently Unreliable
Understanding that language learning models aren’t deterministic ranking machines is crucial. They are probabilistic, synthesizing information from trained data or live searches, providing varying answers influenced by context and intent.
Responses shift depending on the prompts, and identical questions can be phrased in multiple ways, which can lead to challenging questions from your CMO about why certain prompts do not feature your brand despite previous citations. It’s a natural outcome in the evolving landscape of AI-driven visibility.
Even though tracking visibility might be uncertain until user prompting becomes clearer, it remains a valuable aspect of SEO analytics.
If we consider prompt response tracking not as a stable KPI but as a pattern analysis, it becomes something SEOs are already quite familiar with.
Shifting focus from merely checking if you are cited or listed to understanding how responses are structured offers more insightful strategies. Analyze these factors:
- The structure of the response.
- Recurring concepts.
- Key phrases and terms.
- Typical levels of detail involved.
This shift in mindset is imperative.
Traditional SEO vs. AI Pattern Analysis
Traditional SEO involves reverse engineering rankings, whereas AI search encourages us to apply this method by uncovering patterns in AI-generated results.
| Traditional SEO | AI Pattern Analysis |
| Focus on rankings | Understanding concept synthesis |
| Content gap analysis | Topic associations |
| Fixed SERP results | Dynamic AI responses |
| Determined signals | Probability-driven responses |
Through analyzing prompt response patterns, we can dive deep into content-level concept synthesis, beyond the technical framework.
In defining a pattern, look for the themes and recurring topics rather than exact response consistency across outputs.
Each LLM formats its outputs uniquely, yet patterns often emerge within the structures, despite differing retrieval methods and functionalities.
For identifying a pattern:
- It appears in 75% or more outputs.
- Observed across two different AI models, like GPT and Gemini.
- Present across multiple prompts in a consistent way.
The 75% benchmark felt stable enough for my sample sizes to confirm strong patterns rather than randomness. You can adjust this based on your content and context, but this approach has helped me sift consistency from the noise.
For instance, if “pricing transparency” shows up in 9 out of 12 responses and across two models, that indicates semantic relevance—a crucial insight into your content strategy.
The Framework to Implement
Here’s how you can apply this for yourself with a structured framework.
Segment your analysis into the following pattern types:
- Structural patterns.
- Conceptual patterns.
- Entity patterns.
Structural Patterns
Focus here on the organization of responses, identifying aspects like:
- Header and section frequency.
- Consistency in list formatting.
- Order or procedural steps.
- Framing of pros/cons.
- Comparative tables.
- Decision-making frameworks.
These indicators can show how models structure topics.
For example, if your prompt’s outputs repeatedly follow: Definition > Criteria > Tools > Implementation, that’s a structural pattern. Use it to gauge user preferences, although it’s crucial to remember that AI suggestions are just tools to enhance content alignment.
Conceptual Patterns
These vary per topic. They might require deeper analysis to uncover. For example, when focusing on “Best domain registrars,” you might look for:

- Pricing transparency (renewal and purchase).
- Customer service references.
- Inclusion of addons (e.g., WHOIS privacy, free emails).
- Security features.
- Bundling opportunities.
- Transfer processes.
If renewal pricing often emerges in different models and variations, adjust how you frame and discuss it in your content pieces to reflect high relevance.
These patterns offer insight into decision-making associations within AI model frameworks.
Entity Patterns
Examine the appearance of brands, tools, and references in responses, noting:
- Mentions of specific brands.
- Tool or feature associations with brands.
- Category positioning within context.
- Sourced citations and their relevance.
Evaluate how certain features align with specific brands, or notice frequently cited sources. This evaluation helps in assessing brand positioning and opportunities, maybe even within affiliate environments or third-party collaborations.
Constructing Your System
It’s not necessary to invest heavily in prompt-tracking tools, although they simplify the process—I manage with manual tracking, which, despite not being perfect, serves its purpose effectively.
If you’re working solo, adjust the methodology to fit your capacities. This might involve extended tracking periods or lowering pattern consistency thresholds from, say, 75% to a more feasible 60%.
Step 1: Choose and Cluster Your Prompts
Identify three main topics to monitor. Develop 3–5 variations of prompts for each topic.
For example, if one topic is domain registration, my cluster includes:
- How do I register a domain name?
- How can I get a domain name?
- Where can I buy a domain?
Step 2: Create Your Tracking Sheet
To track responses, consider using a simple spreadsheet with columns like this:
| Prompt | LLM | Web Search? (Y/N) | Date | Response | Sources (if applicable) | Is My Brand Mentioned? |
Track LLM versions under the appropriate column to understand when new versions are released and how they impact your data.
Begin capturing this data, then enhance the sheet as needed to include pattern elements. Tools like Claude or ChatGPT can assist in automation, reducing manual labor.
Step 3: Develop a Tracking Plan and Begin Monitoring
To ensure effectiveness, define:
- Which AI models to track.
- Options for search mode—enabled, disabled, or model-decided.
- The prompt frequency to run each test on each model.
- Tracking schedule or frequency.
Engage team members wherever possible and use private modes to reduce contextual biases.
Every week, my team tests each prompt on platforms like ChatGPT and Perplexity, collecting several responses per prompt per model consistently.
Step 4: Conduct Analysis
Once you compile 20-30 responses per prompt, delve into the analysis phase. Select tools to streamline this process effectively.
Identify recurring patterns and link these insights to your site’s relevant pages. Ensure your content addresses discovered themes and questions, and consistently represents the patterns found.
Assess and revise consistently, making this analysis an integral part of your optimization strategy.
Beware of AI Pattern Analysis Pitfalls
AI is inherently probabilistic and not always correct. While it shouldn’t be the sole basis of your strategy, it can offer valuable insights to enhance your playbook.
Risks such as bias in training data, uncertainty in whether search or training data was utilized, and differences in new model launches across LLMs persist.
Use judgment and audience insights to determine when AI responses align with your optimization goals.
Linking Your Strategy to Performance
This is where it gets complex. Though AI responses are notoriously unpredictable, some measurable signals can reflect your content’s impact.
- “Traditional” Metrics: Are you seeing better click rates or improved positions in tools like GSC? Are conversions increasing?
- AI Traffic Monitoring: Analyze AI traffic data from platforms like Adobe or GA4 to note changes on updated pages.
- AI Tracking Tools: While there’s variability here, if utilizing AI visibility tools, they might indicate the effectiveness of your strategy and reflect brand patterns using manual tracking as well.
I recommend experimenting with this manual tracking approach to witness potential brand emergence as a pattern and gain brand visibility.
Begin Examining AI Outputs
Indeed, many unknowns surround LLMs, seemingly changing daily. Yet, one constant remains: these tools provide insights. Leverage any understanding of these responses to enhance your strategies.
Patterns in responses can unravel how subjects are interpreted, how brands appear, and offer guidance on adapting your content strategy.
Inspired by this post on Search Engine Land.
















