In my experience, the open web often feels like the Wild West, especially in recent times. Many creators, myself included, have watched as our hard work is scraped and fed into large language models without any hint of permission.
This situation has become a free-for-all, leaving website owners with almost no means to opt out or safeguard their creative endeavors. There have been attempts to address this, such as Jeremy Howard’s llms.txt initiative. Much like robots.txt helps us manage site crawlers, llms.txt aims to provide guidelines for AI companies’ crawling bots.
However, a promising new protocol is on the horizon, potentially granting site owners like myself more control over how AI firms utilize our content. It looks like this might become part of robots.txt, allowing us to set definitive rules around AI system access and usage.
IETF AI Preferences Working Group
In response to this issue, the Internet Engineering Task Force (IETF) began the AI Preferences Working Group earlier this year in January. Their mission is to craft standardized, machine-readable rules to empower site owners to articulate AI usage preferences for their content.
Since its inception in 1986, the IETF has established core Internet protocols like TCP/IP, HTTP, DNS, and TLS. Now, they’re laying down foundations for the open web’s AI era. Leading this group are co-chairs Mark Nottingham and Suresh Krishnan, joined by figures from Google, Microsoft, Meta, and more.
Of particular interest is Google’s involvement via Gary Illyes, who is part of this working group.
“The AI Preferences Working Group will standardize building blocks that allow for expressing preferences about how content is collected and processed for Artificial Intelligence (AI) model development, deployment, and use.”
What the AI Preferences Group is Proposing
This group aims to deliver new standards that empower site owners to determine how LLM-powered systems can utilize their open web content.
A standard track document detailing a vocabulary to express AI-related preferences, independent of content association methods.
Standard track document(s) that explain how to associate these preferences with content using IETF-defined protocols and formats, for example, Well-Known URIs and HTTP response headers.
A standard approach for reconciling multiple preference expressions.
At the time of writing, nothing is set in stone yet. Early documents, however, provide a sneak peek into potential standards.
This working group published two crucial documents in August.
These documents propose significant updates to the Robots Exclusion Protocol (RFC 9309), suggesting new rules and definitions enabling site owners to specify AI content usage permissions.
How It Might Work
AI systems on the web are categorized and assigned standard labels. Whether a directory will exist for site owners to identify system labels remains unclear.
Currently, the defined labels include:
search: for indexing/discoverability
train-ai: for general AI training
train-genai: for generative AI model training
bots: for all types of automated processing, such as crawling and scraping
For each label, you can set two values:
y to allow
n to disallow.
I found it interesting that these rules can be applied at the folder level and customized for different bots. In robots.txt, they’re implemented using a new Content-Usage field, akin to existing Allow and Disallow fields.
Here’s an example robots.txt that the working group shared in their document:
Explanation Content-Usage: train-ai=n indicates that no content on this domain may be used for training any LLM model, whereas Content-Usage: /ai-ok/ train-ai=y permits model training using content within the /ai-ok/ folder.
Why Does This Matter?
There’s significant buzz about llms.txt within the SEO community and its use alongside robots.txt. Yet, no AI company has confirmed adherence to these guidelines, and Google disregards llms.txt.
Website owners, including myself, crave more explicit control over how AI companies leverage our content—be it for training models or RAG-based responses.
I feel that the IETF’s new standards signify positive progress. With Illyes as a contributing author, I remain optimistic that once finalized, companies like Google will embrace these standards, respecting new robots.txt rules during content scraping.
I’ve been deeply involved in the compelling discussions around AI, especially the intriguing intersection of ‘AI hype meets AI reality.’ Tools like Semrush One and its Enterprise AIO tool have taken center stage, offering invaluable insights into what’s happening inside LLMs. The big questions I often ponder are: How many citations are we capturing and just how many mentions are our brands accumulating?
When this data first emerged, it felt revolutionary. However, it quickly prompted other questions, like ‘What’s the ROI here?’ and ‘How can I integrate this data into my team’s marketing strategy?’ Ensuring that this valuable and fascinating data translates into actionable insights is a challenge I enjoy tackling.
It’s no secret that the data these tools provide is incredibly valuable. But, what steps do I take next? Let’s uncover this journey together.
The Fundamental Challenges of Tracking LLMs
Tracking LLMs can be more challenging than traditional metrics like Google rankings. Google rankings may show where I stand, but ranking doesn’t always correlate with traffic or revenue. Even if I rank highly, an AI Overview could dominate the search, reducing my traffic for a given keyword. I need to ask myself, is this the right traffic for my business goals?
The big difference between traditional SEO rankings and LLM visibility is the straightforward correlation between strong rankings and increased revenue, which is more complex with LLMs. I can easily track user behavior after they land on my site from organic search, but it’s not so clear-cut with LLMs.
SEO effectively drives traffic to my site, allowing me to evaluate the success of my conversion rate optimization (CRO) strategies. However, LLMs operate differently, leaving me with the task of creatively connecting the dots.
The Problem with Methodology
As I dive deeper into using LLM-related data, I realize this approach requires me to step out of my comfort zone as a performance marketer. My usual reliance on direct attribution and data points is shifted toward constructing a narrative that ties LLM visibility to larger brand storytelling.
This method isn’t novel, however. Brand marketers have dealt with indirect metrics since the days of billboard advertising. Still, the shift requires me to create insights from what might seem like fragmented LLM data.
Metrics and Approach to LLM Impact Measurement
Uncovering the true value brought by LLM visibility metrics is a layered and comprehensive process. To do this accurately, I need to understand the wider ecosystem of my organization’s promotional efforts. This understanding allows me to determine the root cause of site traffic or branded searches effectively.
For instance, if a TV ad campaign runs concurrently with optimizing for LLM mentions, analyzing their impact becomes essential. Only with complete awareness of such activities can I identify true causality or correlation.
From here, I find that LLM visibility data is usually just the starting point. It’s unlike traditional SEO insights, which might be more apparent and direct. My task is to delve deeper, probing these data points to uncover richer insights.
The Branded Search of It All
I’ve noticed that brand search provides exceptional insights into LLM performance, offering a rich vein of marketing intelligence. The comparison between two competing chicken wing chains, Buffalo Wild Wings and Wingstop, brightened this understanding for me. While their LLM citations differ, their brand awareness through social media presence offers a clearer picture of market positioning.
Simply examining the branded search traffic showed me how both brands performed similarly on Google, despite their different social media followings. Here lies the heart of utilizing search data creatively to find LLM visibility data strategies.
Rather than merely counting traffic, I am now compelled to consider the number of branded keywords involved, providing a sometimes surprising view on brand awareness and diversity. This approach provides a richer understanding of LLM visibility’s impact.
Direct Traffic: My Trusted LLM Data Companion
I’ve come to see direct traffic as an essential part of my LLM data narrative. Far from being a black hole, direct traffic can often indicate brand awareness and affinity, especially when correlated with LLM visibility metrics. Understanding these correlations allows me to paint a clearer picture of AI’s practical impact on consumer behavior.
For instance, if I compare LG and TCL, LG’s superior direct traffic and increasing momentum in LLM visibility suggest a tangible AI-driven influence, a possibility I must explore through multi-metric analysis.
Considering various metrics together and identifying shared trends offer insight into how LLM visibility might be affecting my brand’s overall recognition and engagement.
Not Just One Metric: Stitching Together LLM Data Stories
Ultimately, it’s about developing a comprehensive data story from LLM visibility insights. This story goes beyond direct KPIs, utilizing various data sources, such as bounce rates and organic traffic, to add depth and relevance to the narrative. Every piece of performance-focused data stands as testimony to the expertise we can bring to LLM visibility.
Total LLM visibility data, when creatively amalgamated with performance data, can transform insights into actionable strategies that align with pragmatic business objectives, showcasing our value in the AI-driven landscape.
Have you ever wondered how to make your products stand out in Google AI Shopping and its AI Mode? I’ve discovered that optimizing feeds, utilizing schema, improving imagery, and crafting conversational Product Detail Page (PDP) content are key strategies to enhance visibility.
Research from Forrester and insights from Blain’s Farm & Fleet have shown me that the real obstacle in AI adoption isn’t the technology itself; it’s how we approach marketing tasks.
Imagine a chocolate company with a cherished, decades-old recipe. They ask an AI tool to identify cost-cutting measures. After several ingredient eliminations and promising margins, sales plummet. Finally, someone tastes the product: “This isn’t even chocolate anymore.”
Aly Blawat from Blain’s Farm & Fleet shared this during a MarTech webinar to highlight why 82% of marketing teams struggle with AI: automation devoid of human insight often exacerbates failure.
According to a Forrester study for Optimove, just 18% of marketers feel at the vanguard of AI adoption, despite 80% anticipating enhanced targeting through AI. Only a quarter have active AI use cases in production.
As Forrester’s Rusty Warner explains, many await software with built-in safeguards before fully embracing AI. Currently, marketing runs like an assembly line, ill-suited for AI’s potential to overhaul workflows.
Positionless Marketing could be the answer. Here, marketers manage everything from data to campaign launches independently, allowing swift action and reserved teamwork for larger initiatives.
Blain’s Farm & Fleet trialed AI for their brand’s cohesive tone across platforms, utilizing Jasper, a protected system. Warner suggests starting small to build confidence, ensuring data integrity for effective AI outcomes.
Successful marketing teams centralize critical data definitions, providing essential signals directly to marketers. Adoption lags not due to the technology, but because organizations aren’t structured to exploit it effectively.
Balancing automation with authentic customer engagement means deploying AI where it can be most beneficial while maintaining a genuine brand experience. At Blain’s Farm & Fleet, human oversight ensures alignment with customer expectations.
The future points toward AI in execution, allowing unique, personalized customer journeys. This shift demands organizations to enhance customer experience expertise across all channels.
For effective AI integration, restructuring marketing workflows and focusing on measurable outcomes are key. The vision includes less manual effort, fewer illustrative meetings, and more tangible customer impact.
By 2026, AI adoption is expected to soar with more vendors providing embedded, coherent AI solutions. Brands like Blain’s Farm & Fleet illustrate the transformation—the right AI application fosters growth, far beyond superficial changes.
Ultimately, AI can’t repair broken systems but amplifies existing conditions. Successful teams must adapt modern workflows and mindset shifts to harness AI’s full potential.
You need to publish more useful content without turning your brand into a production line of polished, interchangeable pages. AI can remove hours of mechanical work, but it can also remove the judgment, specificity, and recognizable point of view that make your content worth choosing.
The answer is not to keep AI out of the workflow. It is to decide where efficiency belongs, where a human must remain accountable, and what every page has to prove before you publish it.
Content quality must serve the reader and the retrieval system
A page can be grammatically clean, keyword-aware, and structurally complete while still failing the reader. It may repeat familiar advice, hide the answer beneath an introduction, make claims it cannot support, or sound as though no identifiable organization chose the words.
In the AI era, useful content has to pass several different tests:
Accuracy: Can you trace every meaningful factual claim to reliable evidence, and have you preserved any necessary limits or uncertainty?
Usefulness: Can the reader make a decision, complete a task, or notice a problem they would otherwise miss?
Specificity: Does the page explain the mechanism, constraint, sequence, example, or trade-off behind its advice?
Distinctiveness: Does it contain a judgment, method, explanation, or framing that reflects what your brand actually knows and believes?
Retrieval clarity: Can a relevant passage stand on its own when a search engine or answer system extracts it from the surrounding page?
Brand coherence: Do the vocabulary, promises, evidence standards, and level of certainty match the rest of your site?
These tests catch different failures. Accurate but generic content is forgettable. Distinctive but unsupported content is risky. Search-ready content that reads like a machine-generated template may earn an impression without earning trust. A page is ready only when it is useful, supportable, recognizable, and easy to interpret.
Keep human judgment where trust is created
The safest division of labor is based on accountability, not on whether a task appears easy. Let AI transform approved material. Keep people responsible for deciding what is true, what matters, what the brand believes, and what the reader should do.
AI is well suited to bounded transformations such as reorganizing notes, proposing outlines, generating headline alternatives, turning a long explanation into a checklist, identifying repeated language, and adapting an approved passage to another format. Those tasks have visible inputs and reviewable outputs.
Human ownership matters most at the points where an error would change meaning or weaken trust:
Selecting the audience, search intent, and decision the page must support.
Choosing evidence and deciding which claims the evidence can genuinely carry.
Contributing subject expertise, exceptions, operational details, and a defensible point of view.
Setting the boundary between established fact, editorial judgment, inference, and uncertainty.
Approving promises about products, outcomes, customers, compliance, or performance.
Accepting final responsibility for the published page and its structured data.
For claims that need proof, do not treat model memory as evidence. A fluent sentence can still be unsupported, overgeneralized, or detached from the conditions that made the original claim true.
Give the model a content contract, not a loose prompt
A prompt that asks for an authoritative SEO page leaves the important decisions unresolved. Before drafting, create a short content contract with fields an editor can inspect:
Reader situation: What has brought this person to the page, and what do they already understand?
Reader job: What should they be able to decide or do after reading?
Primary claim: What is the clearest answer you are prepared to defend?
Evidence packet: Which approved facts, documents, examples, and internal expertise may the draft use?
Brand position: What does your organization believe that a generic overview would not say?
Claim boundaries: What must not be asserted, implied, invented, or generalized?
Voice constraints: Which language patterns should appear, and which should be removed?
Retrieval target: Which question deserves a concise, self-contained answer within the page?
Next action: What useful step should the reader take, even if they never become a customer?
Then run the work in an explicit sequence:
A subject owner approves the reader job, primary claim, evidence, and brand position.
AI proposes an outline in which every section resolves a distinct reader question.
An editor removes sections that exist only to make the page look comprehensive.
AI drafts from the approved contract and evidence packet.
A factual pass checks claims, qualifiers, entity names, citations, and unsupported implications.
A separate brand pass checks judgment, vocabulary, tone, repetition, and generic phrasing.
An optimization pass improves headings, answer units, internal links, metadata, and relevant structured data without changing the approved meaning.
A named human owner approves the visible content and machine-readable representation together.
Separating the passes matters. If one reviewer tries to verify facts, improve voice, shorten sentences, and inspect schema at the same time, the visible polish can distract from a weak claim or an unhelpful answer.
Turn brand voice into an editing system
Authenticity does not depend on a human typing every sentence. It comes from a consistent relationship between what your brand knows, what it believes, what it promises, and what it publishes. AI can help express that relationship, but it cannot invent it responsibly.
Labels such as friendly, expert, bold, or conversational are too subjective to guide a draft. Replace them with observable editorial rules:
Beliefs: Record the principles that shape your recommendations. For example, visible content should answer the question before structured data describes the answer.
Audience contract: State what you owe the reader. This might include explaining constraints, separating evidence from opinion, and never hiding the practical answer behind a sales pitch.
Proof habits: Define when claims need links, examples, named entities, qualifications, or review by a subject expert.
Language choices: List preferred terminology, prohibited hype, acceptable contractions, sentence-length tendencies, and the technical terms that must remain precise.
Boundaries: Document claims the brand will not make, including guarantees, fabricated experience, invented customer stories, and unsupported comparisons.
Approved examples: Save real passages that demonstrate the voice and annotate why they work. A model needs patterns, not just adjectives.
Consider the difference between a generic claim and an owned editorial position.
Generic: AI is transforming content marketing and helping businesses improve efficiency.
Owned: Use AI to compress mechanical work. Keep evidence selection, claim boundaries, and final judgment with an accountable editor.
The second version is not stronger because it sounds more colorful. It makes a decision, draws a boundary, and tells the reader what to do differently. That is the material from which a recognizable brand voice is built.
Use a swap test during editing: if a competitor could publish the paragraph unchanged, it probably lacks an owned insight. Do not add a slogan merely to make it sound branded. Add the missing judgment, mechanism, example, limitation, or operating rule.
Also remove simulated experience. If your organization did not run a test, interview a customer, inspect an account, or observe a result, the draft must not imply that it did. Explain what you know and how you know it. Honest limits are part of brand voice.
Make content easy for people and answer systems to use
Optimization for AI search does not require stripping personality from the page. It requires making the important meaning easy to locate, interpret, and reuse without distortion.
Build important sections as self-contained answer units:
Use a heading that names the actual question or decision.
Answer it in the opening sentence without forcing the reader through background first.
Explain why the answer holds or how the mechanism works.
Name the condition, exception, version, audience, or limitation that changes the advice.
Give the reader a concrete next action.
Link the words carrying an evidence-dependent claim, rather than attaching an unexplained list of links.
The opening answer provides clarity. The mechanism and limitation provide trust. The recommended action is where brand judgment becomes visible. You can therefore write a passage that is both extractable and distinctly yours.
Run a context test on each candidate answer unit. Copy the passage into a blank document and ask:
Is the subject named, or does the passage depend on a vague pronoun?
Can a reader tell whether the statement is a fact, recommendation, definition, or opinion?
Are material conditions and exceptions still present?
Does the passage identify the product, organization, feature, standard, or audience precisely?
Would the passage remain accurate if displayed without the preceding paragraph?
If the answer unit fails outside its original context, revise the language rather than stuffing more keywords into it.
Consistency also matters across the site. Use one canonical name for your organization, products, services, features, and authors. Explain genuine synonyms, but do not rotate terminology simply to create lexical variety. Unnecessary variation makes it harder for a person or system to determine whether two passages refer to the same entity.
Apply the same discipline to JSON-LD and other structured data. Markup should represent the visible page accurately. It should not introduce credentials, ratings, offers, authorship, answers, or relationships that the reader cannot verify in the content. Schema can clarify a strong page; it cannot supply the substance the page is missing.
Finally, use internal links to connect a concise answer with the deeper proof behind it. A summary page can resolve the immediate question, while a supporting page explains the method, terminology, evidence, or implementation. This creates a useful path for readers without forcing every page to become an exhaustive encyclopedia.
Replace output metrics with a publish gate and feedback loop
Traditional quality metrics are not enough for AI-first content. Word count, production volume, grammar checks, and a passing optimization score can describe the artifact or workflow, but they cannot establish that the page is accurate, useful, distinctive, or trusted.
A useful measurement system separates four kinds of signals:
Production signals: Track drafting time, approval loops, substantial rewrites, and where work repeatedly returns to an earlier stage. These reveal workflow efficiency, not content quality by themselves.
Integrity signals: Track unsupported-claim flags, citation gaps, correction requests, entity inconsistencies, and mismatches between visible content and structured data.
Brand signals: Track prohibited language, failed swap tests, unapproved promises, simulated experience, and sections that lack an identifiable editorial position.
Discovery signals: Where your tools can observe them, track the queries that surface the page, branded and non-branded visibility, citations or mentions in answer experiences, and referrals from AI interfaces.
Outcome signals: Match the page to its intended job, such as a completed setup, qualified inquiry, subscription, product comparison, or movement to a deeper supporting page.
Read these signals together. Faster production accompanied by more factual corrections means the workflow moved effort downstream rather than removing it. Strong visibility with weak outcomes may indicate that the page answers the query but does not help with the decision behind it. Good engagement with repeated swap-test failures means the page may be useful while doing little to build brand recognition.
A composite quality score can help you prioritize review, but it should not own the publishing decision. Use a simple editorial gate:
Block: A material claim lacks evidence, the page invents experience, a required limitation is missing, an entity is misrepresented, or structured data asserts something the visible page does not support.
Revise: The answer is buried, advice remains generic, sections repeat one another, the next action is unclear, or the language fails the brand’s documented rules.
Publish: The page answers a real reader need, important claims are supportable, brand judgment is visible, answer units survive the context test, and a named owner accepts responsibility.
After publication, feed what you learn back into the system. Log corrections with their causes. Add strong and weak passages to the annotated voice examples. Update the content contract when reviewers keep fixing the same omission. Revisit important pages when the offer, evidence, entity information, or reader decision changes.
Key takeaways
Use AI for bounded, reviewable transformations; keep people accountable for evidence, judgment, promises, and approval.
Define brand voice through beliefs, proof habits, language rules, boundaries, and annotated examples rather than vague tone adjectives.
Write self-contained answer units that give a direct answer, explain the mechanism, preserve limitations, and recommend a useful action.
Keep entity language, visible content, internal links, and structured data consistent.
Measure production efficiency separately from integrity, brand distinctiveness, discovery, and reader outcomes.
Block publication when a material claim, implied experience, or machine-readable assertion cannot be supported.
Start with one commercially important page. Write its content contract, mark every evidence-dependent claim, run the swap and context tests, and compare its structured data with what a reader can actually see. The weaknesses you find will tell you exactly which rules your wider AI content workflow needs next.
