Tag: AI Discovery

I have watched the debate around llms.txt become one of the most polarized conversations in web optimization.

Some people treat llms.txt as essential infrastructure for AI discovery. Others, especially longtime SEO practitioners, see it as speculative theater. Platform tools are starting to flag missing llms.txt files as site issues, yet server logs still show that AI crawlers rarely request them.

Google even appeared to adopt it. Sort of. In December, Google added llms.txt files across many developer and documentation sites.

At first, the signal looked obvious to me: if the company behind the sitemap standard was implementing llms.txt, maybe the file really mattered.

Then Google removed it from its Search developer docs within 24 hours.

Google’s John Mueller said the change came from a sitewide CMS update that many content teams didn’t realize was happening. When asked why the files still exist on other Google properties, Mueller said they aren’t “findable by default because they’re not at the top-level” and “it’s safe to assume they’re there for other purposes,” not discovery.

The llms.txt research

I wanted data, not another debate.

So I tracked llms.txt adoption across 10 sites in finance, B2B SaaS, ecommerce, insurance, and pet care. I looked at the 90 days before implementation and the 90 days after.

I measured AI crawl frequency, traffic from ChatGPT, Claude, Perplexity, and Gemini, and the other changes each site made during the same window.

Here is what I found:

• Two of the 10 sites saw AI traffic increases of 12.5% and 25%, but llms.txt was not the cause.
• Eight sites saw no measurable change.
• One site declined by 19.7%.

The 2 ‘success’ stories weren’t about the file

The Neobank: 25% growth

One digital banking platform implemented llms.txt early in Q3 2025. Ninety days later, its AI traffic was up 25%.

That sounds compelling until I looked at what else happened during the same period.

• The company ran a PR campaign around its banking license and earned coverage in major national publications.
• It restructured product pages with extractable comparison tables for interest rates, fees, and minimums.
• It published 12 new FAQ pages optimized for extraction.
• It rebuilt its resource center with new banking information and concepts.
• It fixed technical SEO issues, including header structure problems.

When a company earns Bloomberg coverage in the same month it launches optimized content and fixes crawl errors, I cannot isolate llms.txt as the growth driver.

The B2B SaaS platform: 12.5% growth

A workflow automation company saw AI traffic jump 12.5% two weeks after implementing llms.txt.

The timing looked perfect. It would be easy to call the case closed. But the surrounding context told a different story.

Three weeks earlier, the company had published 27 downloadable AI templates covering project management frameworks, financial models, and workflow planners. These were functional tools, not ordinary content marketing assets, and they drove the engagement behind the spike.

Google organic traffic to those templates rose 18% during the same period and kept climbing throughout the 90 days I measured.

Search engines and AI models surfaced the templates because they solved real problems and created an entirely new site section. They did not surface them simply because the URLs appeared in an llms.txt file.

The 8 sites where nothing happened after uploading llms.txt

Eight sites saw no measurable change after adding llms.txt. One of them declined by 19.7%.

The decline came from an insurance site that implemented llms.txt in early September. Based on the data, the drop likely had nothing to do with the file.

The same pattern appeared across all traffic channels. Llms.txt did not prevent the decline, and it did not create any visible advantage.

The other seven sites, which included ecommerce brands in pet supplies, home goods, and fashion, plus B2B SaaS, finance, and pet care sites, used llms.txt to document their best existing content. That content included product pages, case studies, API docs, and buying guides.

Ninety days later, nothing changed. Traffic stayed flat. Crawl frequency was identical. The content was already indexed and discoverable, and the file did not change that.

The pattern was clear: sites that launched new, functional content saw gains. Sites that only documented existing content saw no gains.

Why the disconnect?

No major LLM provider has officially committed to parsing llms.txt. Not OpenAI. Not Anthropic. Not Google. Not Meta.

Google’s Mueller put it plainly:

• “None of the AI services have said they’re using llms.txt, and you can tell when you look at your server logs that they don’t even check for it.”

That is the reality I saw in the data. The file exists. The advocacy exists. But platform adoption does not show meaningful use yet.

The token efficiency argument and its limits

The strongest case for llms.txt is efficiency. Markdown can save time and tokens when AI agents parse documentation. It gives agents clean structure instead of forcing them through complex HTML, navigation, ads, and JavaScript.

Vercel says 10% of its signups come from ChatGPT. Its llms.txt includes contextual API descriptions that help agents decide what to fetch.

That matters, but mostly for developer tools and API documentation. If your audience uses AI coding assistants like Cursor or GitHub Copilot to interact with your product, token efficiency can improve integration.

For ecommerce brands selling pet supplies, insurance companies explaining coverage, or B2B SaaS companies targeting nontechnical buyers, token efficiency does not automatically translate into traffic.

llms.txt is a sitemap, not a strategy

The closest comparison I can make is a sitemap.

Sitemaps are useful infrastructure. They help search engines discover and index content more efficiently. But I would not credit traffic growth to simply adding a sitemap. The sitemap documents what exists; the content drives discovery.

Llms.txt works in a similar way. It may help AI models parse a site more efficiently if they choose to use it, but it does not make the content more useful, authoritative, or likely to answer user queries.

In my analysis, the sites that grew did so because they:

• Created functional assets such as downloadable templates, comparison tables, and structured data.
• Earned external visibility through press and backlinks.
• Fixed technical barriers such as crawl and indexing issues.
• Published content optimized for extraction, including FAQs and structured comparisons.

Llms.txt documented those efforts. It did not drive them.

What actually works

The two successful sites showed me what actually matters.

• Create functional, extractable assets. The SaaS platform built 27 downloadable templates that users could deploy immediately. AI models surfaced them because they solved real problems, not because they appeared in a markdown file.
• Structure content for extraction. The neobank rebuilt product pages with comparison tables for interest rates, fees, and account minimums. That is data AI models can pull directly into answers without heavy interpretation.
• Fix technical barriers first. The neobank fixed crawl errors that had blocked content for months. If AI models cannot access your content, no amount of documentation will help.
• Earn external validation. Coverage from Bloomberg and other major publications drove referral traffic, branded searches, and likely influenced how AI models assessed authority.
• Optimize for user intent. Both sites answered specific queries, such as “best project management templates” and “how do [brand] interest rates compare?” Models surface content that maps to what users ask, not content that is merely well documented.

None of this requires llms.txt. All of it can drive results.

Should you implement an llms.txt file?

If you run a developer tool and AI coding assistants are a primary distribution channel, I would implement llms.txt. In that context, token efficiency matters because your audience is already using agents to work with documentation.

For everyone else, I would treat llms.txt like a sitemap: useful infrastructure, not a growth lever.

It is good practice to have. It likely will not hurt. But the hour spent implementing llms.txt is often better spent restructuring product pages with extractable data, publishing functional assets, fixing technical SEO issues, creating FAQ content, or earning press coverage.

Those tactics have shown real ROI in AI discovery. Llms.txt has not, at least not yet.

The lesson I take from this is not that llms.txt is bad. It is that we are reaching for control in a system where the rules are still being written. Llms.txt offers comfort because it is concrete, actionable, and familiar. It looks like the web standards we already understand.

But looking like infrastructure is not the same as functioning like infrastructure.

My focus would stay on what is already working:

• Create useful content.
• Structure it for extraction.
• Make it technically accessible.
• Earn external validation.

Platforms and formats will change. The fundamentals will not.

Inspired by this post on Search Engine Land.