I’ve recently discovered that Google has introduced a new feature in Chrome Lighthouse to check for llms.txt files. Though Google mentions that llms.txt isn’t necessary for AI search visibility, Lighthouse has started flagging sites based on their presence.
Google’s latest Lighthouse audits, under the “Agentic Browsing” category, now focus on a site’s usability for machine interaction. I find this interesting as it aligns with Google’s push towards better machine readability.
The new audits are part of Chrome’s evolving “Agentic Browsing” features, which analyze if sites are prepared for automated interaction. This concept came soon after Google issued guidance on AI search optimization, debunking the necessity of llms.txt files in their new guide on generative AI features.
What Lighthouse Evaluates Now. Lighthouse’s Agentic Browsing tests focus on how well my site is built for machine interactions, incorporating various deterministic audits as per Google’s documentation. These checks include:
– WebMCP integration.
– Accessibility tree integrity.
– Layout stability through CLS.
– Presence of an llms.txt file.
These audits help ensure that there’s a machine-readable summary at the site’s domain root. Google explains that without llms.txt, agents might take longer to understand a site’s main structure.
The impact of these audits doesn’t translate into a traditional Lighthouse score but into a fractional pass ratio related to agentic readiness signals.
The Tension. Interestingly, while these audits don’t directly affect SEO rankings, their mention in Google’s readiness checks could make SEOs reconsider their stance on llms.txt files.
Agentic Engine Optimization. Google’s approach aligns with insights shared by Addy Osmani from Google Cloud AI about Agentic Engine Optimization. Osmani emphasizes creating web content that is semantically structured, token-efficient, and easy for AI to process.
SEO vs. llms.txt. According to Google, creating llms.txt or similar files isn’t necessary for AI search success, as outlined in the guide on Mythbusting generative AI search. The AI systems can discover, crawl, and index a variety of file types encountered on the internet.
John Mueller from Google responded to concerns about the role of llms.txt in a discussion with Lily Ray on Bluesky, stating that the use of these files is more for functionality and not directly linked to search engine optimization.
Google’s Take on AI Agents. Besides llms.txt, Google’s Lighthouse guidelines place strong emphasis on accessibility and interface stability. The insight I gained is that AI agents heavily rely on the accessibility tree as their core data model, focusing on integrity and proper layout.
Ultimately, while Google indicates llms.txt isn’t needed for search, including such files might be beneficial for adapting to Google’s evolving tools that prioritize machine readability.
Further Exploration.
– Meet llms.txt, a proposed standard for AI website content crawling
– llms.txt isn’t robots.txt: It’s a treasure map for AI
– Does llms.txt matter? We tracked 10 sites to find out
Inspired by this post on Search Engine Land.
