Have you ever wondered how all those Claude bots from Anthropic handle your site’s data? Well, I’ve delved into their latest update, which offers insights into their AI training, real-time queries, and what happens when you choose to block them.
Anthropic recently enhanced their crawler documentation, providing clarity on how Claude bots interact with websites and how you can regain control by blocking them.
Why should you care? If you’re like me and manage content, you’ll want to manage how AI systems utilize your work. Anthropic smartly divides bots into training crawlers, user-initiated fetches, and search indexers. Blocking just one won’t impact the others, so make informed choices based on visibility and training implications.
Let’s meet the robots: Anthropic employs three unique user agents. First up, ClaudeBot gathers public online content for training their AI models. Blocking it means your site’s content won’t be in future AI datasets.
Next, there’s Claude-User, which fetches pages when someone asks Claude a question necessitating site access. Block this bot and lose out on visibility in user-driven response queries.
Finally, Claude-SearchBot improves search results by indexing. If you decide to block it, it may affect your content’s visibility and accuracy in Claude-enhanced search responses.
Curious about blocking these bots? They comply with standard robots.txt directives, including “Disallow” and “Crawl-delay”. To block a bot site-wide, use:
User-agent: ClaudeBot Disallow: /
Bear in mind, each bot and subdomain you wish to limit needs its own directive. Be cautious with IP blocking; these bots operate via public cloud IPs, which might interfere with robots.txt access, and IP details aren’t disclosed by Anthropic.
The landscape of AI is rapidly shifting in 2026. I’ve noticed that AI models are losing their once shared data access, resulting in fragmented and less cohesive answers.
This change is primarily due to the surge in platform-controlled data, which is significantly altering how visibility and search functions within AI systems. It’s intriguing to see how these developments are reshaping the way we interact with and trust AI-driven responses.
I recently came across fascinating research revealing how diverse AI platforms like ChatGPT, Google AI, and Perplexity cite their sources. It’s intriguing to see how each platform approaches sourcing information and the implications for their visibility.
The study highlights substantial differences in citation patterns among these major AI players. This variation in sourcing methods significantly affects how each platform is perceived in terms of reliability and authority.
Understanding these citation patterns can offer valuable insights into the competitive landscape of AI visibility. As we explore this further, it becomes clear why recognizing these differences is crucial for anyone interested in AI optimization.
I’ve noticed how search is evolving far beyond the typical blue-links framework. Now, discovery often happens within AI-generated answers—whether it’s Google AI Overviews, ChatGPT, or other LLM-driven platforms. It’s clear to me that visibility is no longer just about rankings, and influence doesn’t always lead to a click.
Traditional SEO metrics like rankings, impressions, and CTR seem to fall short as search becomes more recommendation-driven and attribution becomes increasingly opaque. Clearly, a new measurement layer for SEO is needed.
This is where LLM consistency and recommendation share (LCRS) steps in. It helps measure how reliably and competitively my brand appears in AI-generated responses. It’s a modern equivalent to keyword tracking, tailored for the LLM era.
Why traditional SEO KPIs are no longer enough
Traditional SEO metrics worked well when visibility was tied directly to ranking positions and user interaction pivoted on clicks. This relationship weakens in LLM-mediated searches. Even if my page ranks at the top, it may never appear in an AI-generated answer.
LLMs might favor another source with lower traditional visibility, exposing a flaw in conventional traffic attribution. Here, brand influence might occur without a measurably corresponding website visit. The impact exists but isn’t reflected in the traditional analytics landscape.
At the heart of this change is something that traditional SEO KPIs were not developed to handle:
Being indexed means my content is available for retrieval.
Being cited means it serves as a valuable source.
Being recommended highlights my brand as an active solution or answer.
Traditional SEO analytics often stop at indexing and ranking. However, in a world dominated by LLM-driven search, the true competitive edge lies in recommendation—a dimension current KPIs struggle to quantify. This is where the gap between influence and measurement creates a space for new performance metrics.
LCRS: A KPI for the LLM-driven search era
With LLM consistency and recommendation share, I can gauge how reliably my brand surfaces and is recommended by LLMs during search and discovery processes.
LCRS answers a crucial question that traditional SEO metrics can’t: When users look to LLMs for guidance, how often and consistently is my brand part of the conversation?
It evaluates my visibility across three dimensions:
Prompt variation: Different user ways of asking the same question.
Platforms: Various LLM-driven interfaces.
Time: Consistent appearances over time, not just one-shot mentions.
LCRS is less about isolated citations and more about establishing a repeatable, comparable presence, enabling me to benchmark against competitors and track changes.
Although it’s not a replacement for established SEO KPIs, LCRS enhances them by addressing zero-click search scenarios where recommendations determine visibility.
Breaking down LCRS: The two components
LCRS comprises two primary elements: LLM consistency and recommendation share.
LLM consistency
In LCRS, consistency measures how reliably my brand appears across similar LLM responses. High consistency means my brand surfaces across numerous, semantically similar prompts rather than relying on a single high-performing query.
Considerations like prompt variability, temporal variability, and platform variability come into play. Consistency reflects durable relevance beyond transitory exposure.
Recommendation share
While consistency focuses on repeatability, recommendation share assesses competitive presence. It examines how frequently LLMs recommend my brand relative to others in the same category.
Not all appearances count as recommendations; it’s about how often my brand is positioned as a primary choice against competitors, reflecting the portion of recommendation space occupied.
How to measure LCRS in practice
To effectively measure LCRS, a structured approach is necessary, one that replaces anecdotal observations with repeatable sampling reflective of actual user interactions.
1. Select prompts
I start with choosing prompts representing my category, ensuring they include variations in phrasing to capture natural language nuances.
2. Confirm tracking
The choice between brand-level and category-level tracking hinges on focus. Most insightful at the category level, LCRS shows which brands LLMs choose to highlight.
3. Execute prompts and collect data
Since managing data volumes is a challenge, I rely on programmatically executing prompts and parsing responses to identify which brands are recommended.
4. Analyze the results
Automated data capturing is key, though human review is crucial for interpreting nuanced information. Tracking analysis over time is essential for stable directional signals.
Use cases: When LCRS is especially valuable
LCRS is particularly valuable in environments where synthesized answers shape decisions. In marketplaces, SaaS, YMYL industries, and comparison searches, LLMs significantly influence visibility.
Limitations and caveats of LCRS
LCRS offers directional insight rather than definitive certainty, given LLMs’ non-deterministic nature. Short-term volatility is expected, so evaluating trends over time is vital.
This metric isn’t a replacement for traditional analytics but complements them by addressing influence areas without direct attribution.
What LCRS signals about the future of SEO
More than a ranking tool, LCRS signals a shift toward brand presence engineering in the LLM-driven discovery space. Brand authority is becoming crucial, with successful SEOs adapting to optimize for retrievability, clarity, and trust.
The shift from position to presence
As LLM-driven search reshapes discovery, expanding from ranking positions to presence and recommendation is crucial. LCRS allows me to explore this gap and complement existing performance metrics for a comprehensive visibility strategy.
My journey with LCRS shows that adapting SEO strategies for evolving landscapes boosts both visibility and influence within LLM-driven search experiences.
As I delve into the world of AI searches for wearable technology, I’ve noticed a fascinating trend. It turns out that trusted third-party sites are more frequently favored over brand websites. This piqued my curiosity, and I wanted to dig deeper into these patterns and uncover how one can achieve AI visibility.
One of the key aspects that stood out is the consistency in how certain domains are cited across AI searches. These sites have established a level of trust and authority that AI algorithms consistently recognize. As I’m navigating through this data, I’m exploring the most frequently cited domains in this realm and the trust patterns they demonstrate.
Gaining AI visibility isn’t just about being present; it’s about earning trust and authority. By understanding these patterns, I feel more equipped to help others and myself in enhancing the visibility of our wearable tech offerings.
I’ve been thinking a lot about the key performance indicators (KPIs) for AI search, and it’s time to shift our focus a bit.
Lately, I’ve noticed many SEO experts on platforms like LinkedIn and during conferences discussing the idea of “ranking No. 1 on ChatGPT,” equating it to securing the top spot on Google.
On Google, being first is often like striking gold.
However, this isn’t necessarily true with AI-generated responses, primarily because these responses are subject to constant change.
Our research indicates that AI users evaluate an average of 3.7 businesses before making a choice.
Thus, appearing first in ChatGPT’s results isn’t as crucial as it is in Google’s search results.
Given this scenario, our AI strategy should prioritize “being part of the consideration set” over being the first mention and focus on what AI communicates about us.
In the past months, my team has devoted over 100 hours observing how people use ChatGPT and Google’s AI Mode for finding services.
What became clear quickly is that user behavior on AI search platforms is distinctively different from that on Google, beyond just the use of natural language versus keyword searches.
In both Google’s AI Mode and ChatGPT, users typically consider 3.7 businesses from the results shown.
This significantly affects the importance of being the top result and elevates the value of other positions, as 75% of users also review businesses listed from positions 2 to 8.
Ultimately, what drives conversions isn’t solely your position in that list.
These aren’t traditional rankings; they’re more akin to recommendations which might change in order or format, underscoring AI’s probabilistic nature.
AI chat interfaces allow users to scan and assess more options feasibly than Google search results do.
If a user is evaluating fractional CMO options, it’s more work through Google Search than ChatGPT.
In Google’s results for “fractional CMO,” only two appear above the fold, each requiring click-through to view their full details.
Contrast that with ChatGPT, where the model offers eight options with concise descriptions.
This convenience makes it easier to make informed choices.
We need to ensure that what the model says about us aligns with our message.
Many marketers prioritize rankings and traffic but overlook messaging and positioning.
Our study shows approximately 60% of users finalize their decisions based solely on AI responses without further exploring the business’s website or using Google.
To enhance conversion, we must deliver the correct message and ensure the AI conveys it accurately.
For instance, even if Dr. Lanciano is the best in glaucoma care, if the AI promotes Ravi D. Goel and Bannett Eye Centers, users might lean towards them if that suits their needs.
This reaffirms that appearing last doesn’t negate conversion opportunities if the AI message resonates well, unlike traditional search.
Visibility alone doesn’t bring in revenue; conversions do, and these happen when prospects perceive your solution as a fit.
AI outputs can be wildly inconsistent, and Rand Fishkin recently spotlighted this issue. His research revealed that AI tools produce varied brand recommendations, which highlights the need for a deeper understanding beyond ranking positions.
After reading his work, I realized the solution is rooted in something I’ve been developing for years – building consistent visibility through confidence and corroboration.
Fishkin’s data showed that AI systems are confidence engines. They draw results based on confidence levels, which explains the inconsistency in output. It’s a problem when there’s low confidence, but once AI systems are confident, they provide consistent recommendations.
The journey to AI confidence involves several stages, and understanding this process can fundamentally change how brands approach AI visibility.
Take the entity home as an example. It’s the foundation of AI interpretation of your brand. Confidence also builds when third-party data aligns with your own narrative. Brands that manage this well don’t just appear in AI recommendations; they dominate them.
There’s a method behind all this that I’ve formalized and even filed for patenting. It’s a complex system of strategies but starts with ensuring that your brand’s digital footprint aligns perfectly with high-authority sources.
Fishkin’s work confirms the importance of AI visibility, a subject I’ve been tracking and developing solutions for over the last decade. It bridges a significant gap in understanding how brands can leverage AI for long-term authority and presence.
As I dive into the ever-evolving world of AI search engines, I find myself asking: which one should my brand optimize for first? The options are plentiful, with ChatGPT, Google AI Overviews, Perplexity, Bing, and others vying for attention. The goal is clear: prioritize AI visibility leading into 2026, but the path there is not so straightforward.
Each of these AI platforms offers unique features and potential benefits that can cater to different business needs. It’s crucial for me to assess their capabilities and align them with my brand’s strategic objectives. Whether it’s the conversational prowess of ChatGPT or the data-rich insights from Google AI Overviews, the choice has to drive brand value.
In the process of optimization, understanding the nuances of each platform helps to leverage their full potential. By comparing these engines, I can tailor my approach, ensuring my brand stays ahead in AI visibility, making informed decisions today that will resonate in the future.
A quick five-minute video can offer more data to a large language model than many blog posts. Here’s how I can enhance my brand’s visibility for AI data retrieval.
With OpenAI’s significant deal with Disney, web scraping is undergoing a transformation. This agreement lets OpenAI employ high-fidelity, human-verified cinematic content to minimize AI inaccuracies.
These opportunities enhance my brand’s visibility and recognition, as AI models crave high-quality data. Video becomes a crucial asset for my brand in this evolving landscape.
Here’s why video is becoming the AI’s truth source and how I can leverage it to defend my brand’s identity.
Brand drift in AI occurs when an AI doesn’t have specific data about my brand, leading it to piece together my brand’s story from generalized information.
This interpolation risks creating misleading brand narratives. Imagine a situation where an AI inaccurately describes my SaaS company’s product features because it lacks precise data.
Streamer.bot faced a similar issue, with AI-generated instructions that were confidently incorrect, creating unnecessary confusion and workload.
Even local businesses are affected. A restaurant owner reported repeated inaccuracies shared by Google AI about their menu in an article.
Providing a canonical truth source, like video, prevents AI from distorting my brand’s message.
Authoritative videos carry significant semantic value, offering detailed transcripts and visual proof that establish a solid truth source, helping avoid misinformation from any other platforms.
Videos pack high data with nuance, offering multiple layers of communication through visuals, sound, and text.
Studios such as Berlin-based Impolite produce high-quality videos to help brands retain their identity, preventing brand drift by offering rich data sources for AI.
For instance, Karman’s “The Space That Makes Us Human” project showcases expert-led video that serves as an authentic truth source for brands.
Authenticity now acts as a crucial technical signal. Verification ensures that AI models can trust the provenance of a video.
Real-world footage is the ultimate high-trust data source. AI-generated videos typically lack the real-world’s dynamic intricacies.
Organizations like the Coalition for Content Provenance and Authenticity (C2PA) and the Content Authenticity Initiative (CAI) enhance digital content transparency.
These entities allow brands to digitally sign videos, establishing a trustworthy indicator for AI models versus unsigned content.
Similarly, I can understand more about media verification, establishing an unbroken chain of evidence from creation to consumption.
On LinkedIn, a “CR” mark on media indicates its origin and editing history, boosting content authority and authenticity.
Google’s integration of C2PA signals ensures AI-related policies are reflected in search and ads, maintaining accurate representation and disclosure.
In content marketing, adopting C2PA helps me safeguard against misinformation, acting as a quality assurance measure.
If necessary, I can utilize Sony’s camera authenticity solutions to embed real-time digital signatures in media, proving it’s genuine and trustworthy.
C2PA-compliant editing tools allow me to create a manifest detailing all edits and tools used, preserving the content’s integrity.
A cryptographic seal verifies the content’s integrity, alerting AI to broken data chains, ensuring only accurate information is spotlighted.
Given the content overload today, traditional verification methods struggle, but verified subject matter experts (SMEs) stand out as credible sources online.
By pairing expert insights with video evidence, brands provide AI with authentic, non-replicable authority that audiences trust.
Incorporating video as central content captures nuanced details, giving birth to high-quality content across various media platforms.
Repurposing video into text, images, audio, and social media content builds an authority loop, increasing the probability of data retrieval by AI models.
I should predict where AI might misrepresent my brand and utilize verified expert voices and video documentation to address potential misinformation.
It remains vital for me to focus on context over mere compliance in brand building through high-fidelity, cryptographically signed video, safeguarding identity and authenticity.
The mandate is simple: Record reality. Ensuring I provide a verifiable video record prevents AI from creating false narratives about my brand.
Every day, millions turn to ChatGPT for answers, but have you noticed your brand isn’t included in those results? I’ve been there, wondering why my brand isn’t gaining visibility and how to change that. If you’re like me and want to understand what’s happening, I’ve gathered the seven main reasons why ChatGPT might be ignoring your brand.
Understanding these reasons is the first step to making a change. You’ll learn specific steps to enhance your visibility in AI searches, and I can tell you from experience, it’s worth the effort.
Perhaps you’re wondering: what can I do to ensure my brand stands out? Don’t worry, I’m here to guide you through actionable strategies for gaining prominence in AI search results.
