I’ve been following the lively debate around creating separate markdown pages for LLMs, and it appears that both Google and Bing are advising against this approach.
Recently, I noticed that representatives from Google Search and Bing Search have specifically recommended not to create separate markdown (.md) pages designed exclusively for LLMs. This practice involves presenting different content to the LLMs compared to what users see, which can be considered a form of cloaking—a direct violation of Google’s policies.
The question arose when Lily Ray inquired on Bluesky about the prevalence of creating markdown or JSON pages targeted at bots.
- “Not sure if you can answer, but starting to hear a lot about creating separate markdown / JSON pages for LLMs and serving those URLs to bots.”
Google’s stance, as explained by John Mueller, is clear. He replied to Lily’s query saying that LLMs have always interacted with standard web pages and don’t require separate markdown pages.
- “I’m not aware of anything in that regard. In my POV, LLMs have trained on—read & parsed—normal web pages since the beginning, it seems a given that they have no problems dealing with HTML. Why would they want to see a page that no user sees? And, if they check for equivalence, why not use HTML?”
John Mueller even criticized the whole idea, stating:
- “Converting pages to markdown is such a stupid idea. Did you know LLMs can read images? WHY NOT TURN YOUR WHOLE SITE INTO AN IMAGE?” Of course, converting your entire site to a markdown format is an extreme measure.
I’ve collected many of John Mueller’s remarks on this topic, which you can find here.
Bing’s perspective is shared by Fabrice Canel from Microsoft Bing, who suggested that creating duplicate, non-user content isn’t effective.
- “Lily: really want to double crawl load? We’ll crawl anyway to check similarity. Non-user versions (crawlable AJAX and like) are often neglected, broken. Humans eyes help fixing people and bot-viewed content. We like Schema in pages. AI makes us great at understanding web pages. Less is more in SEO!”
Why this matters to us: Many of us are tempted by shortcuts to improve search engine performance. Yet, these shortcuts often backfire or yield short-lived benefits. As Lily Ray remarked on LinkedIn, managing duplicate and differing content for bots violates established search engine policies.
Lily Ray’s thoughts on this are clear:
- “I’ve had concerns the entire time about managing duplicate content and serving different content to crawlers than to humans, which I understand might be useful for AI search but directly violates search engines’ longstanding policies about this (basically cloaking).”
Inspired by this post on Search Engine Land.
Leave a Reply