AI Crawler Discoverability: Fix Technical Access First

You can publish authoritative content, add structured data, and refine every answer on the page, yet remain invisible to an AI crawler for a simpler reason: the crawler cannot find or fetch the URL.

The practical fix is to audit the entire access path, not another layer of copy. Start with the links a crawler encounters, the response your server returns, and the controls between the crawler and the content. Once those work, optimization has something to operate on.

Separate discoverability from ranking and citation

AI visibility is often discussed as if it were one outcome. It is actually a sequence of technical conditions:

1. Discovery: The crawler learns that a URL exists.
2. Access: The crawler is permitted to request it and receives a usable response.
3. Rendering or retrieval: The meaningful page content and links are available in a form the crawler can process.
4. Extraction: The system can distinguish the main content, entities, claims, and relationships.
5. Selection: A search or AI system decides whether the material is relevant and trustworthy enough to use.

A failure at an early stage makes work at a later stage irrelevant. Schema cannot rescue a page whose URL is never discovered. Better prose cannot overcome a blocked request. An llms.txt file cannot repair a navigation system that hides destinations behind JavaScript events.

This distinction also prevents a common reporting mistake. A crawler request in your server logs proves access, not inclusion, ranking, citation, or referral traffic. Conversely, a page that loads in your browser has not necessarily passed a crawler test. Your browser may execute JavaScript, retain cookies, pass a bot challenge, or carry an authenticated session that the crawler does not have.

Define the problem precisely before changing anything. If no request reaches the server, investigate discovery. If requests receive blocks, redirects, challenges, or errors, investigate access. If requests succeed but important text and links are absent from the returned HTML, investigate rendering and delivery.

Make important URLs discoverable through real HTML links

The most consequential crawl problem may be hiding in a navigation component that looks perfectly normal to a person. A clickable card, menu item, or button can move a browser to another page without exposing an ordinary HTML anchor and href in the initial markup. The interface works, but the crawl path does not.

The size of that gap is not theoretical. In one documented implementation, six AI crawlers discovered none of 293 pages linked through JavaScript. After those destinations were converted to ordinary HTML links, GPTBot crawled 250 of the pages in less than three hours. That is one observed case, not a universal crawl-rate benchmark, but it isolates an important mechanism: changing the discovery path can change crawler coverage without changing the underlying content.

For every page you want AI systems to encounter, provide a stable URL through an anchor element with a genuine href. JavaScript can still enhance the interaction. It should not be the only place where the destination exists.

• Use ordinary anchor links for navigation items, topic cards, article lists, pagination, related content, and calls to supporting resources.
• Place the destination in the href attribute rather than only in an onclick handler, router callback, or data attribute.
• Return meaningful navigation in the server-delivered HTML. Do not require a click, scroll, consent interaction, or client-side state change before links appear.
• Give each important resource a stable URL. A view that exists only as transient application state is difficult to discover and reference independently.
• Keep important pages within a coherent internal-link structure. Orphan URLs should not depend on someone already knowing their addresses.

A sitemap can provide another route to a URL, but it should be a supplement rather than compensation for broken internal navigation. Not every crawler uses every discovery mechanism in the same way. HTML links also communicate how pages relate to one another, which a detached list of URLs does not fully replace.

Inspect the raw response rather than relying only on the rendered browser view. Choose a hub page, retrieve its original HTML without an authenticated session, and search for the exact destinations you expect a crawler to follow. If the URLs appear only after scripts run, treat that as a crawl risk even when a modern browser renders the experience correctly.

Treat llms.txt as a signpost, not an access-control system

An llms.txt file can organize selected resources for systems that choose to consume it. It does not replace crawlable navigation, and it is not automatically the correct place to permit or prohibit crawler access.

That distinction matters because publishing the file is not the same as making it useful. Among 584,107 llms.txt files read through Common Crawl, 68.27% matched plugin-generated output and 22.56% contained no links. Another 32 attempted to block a crawler through a file that the crawler did not use for that purpose. Those measurements describe one analyzed corpus, but they expose three recurring implementation errors: unattended template output, empty files, and misplaced access rules.

If you publish llms.txt, give it a defined job. Use it to point toward a small, intentional set of authoritative resources. Check that every listed URL resolves to the intended public page, remove placeholders, and regenerate the file when canonical URLs change. A file produced by a plugin still needs editorial ownership; automation does not know which pages best represent your business or which obsolete URLs should disappear.

If your real goal is access control, configure the mechanism the named crawler says it observes. That commonly involves crawler-specific robots.txt rules plus the behavior of your web server, CDN, firewall, and bot-management layer. Do not assume a directive is effective merely because it contains a crawler name. Confirm that the directive belongs in that file and then verify the resulting request behavior.

Keep the roles clear:

• HTML links expose navigable relationships between pages.
• Sitemaps provide a separate inventory of URLs for crawlers that consume them.
• llms.txt can offer an optional, curated map for systems that support it.
• robots.txt and crawler-specific controls express crawl permissions when the crawler recognizes those controls.
• Server, CDN, firewall, and authentication rules determine whether the request actually reaches usable content.

None of these layers proves that a page will be selected or cited. Their job is to remove preventable technical failures before relevance and authority are evaluated.

Audit the request path from a crawler’s point of view

A useful audit follows the same path a crawler must follow. Do not begin with every URL on the site. Start with one public entry page, one category or topic hub, and a small set of representative deep pages. Include the templates that generate most of your important content, because one template repair can restore access to many URLs.

1. Build a representative test set. Include the homepage or another likely entry point, a hub page, a standard content page, and any valuable page type with interactive navigation.
2. Retrieve each URL as an unauthenticated client. Record the final URL, HTTP status, content type, redirect path, and response body. A successful-looking page in a normal browser is not enough.
3. Check the returned HTML. Confirm that the main text, title, canonical destination, and links to the next important pages are present without waiting for JavaScript.
4. Trace internal discovery. Starting at the entry page, verify that ordinary href links lead to hubs and then to deep pages. Flag every required destination that exists only after an interaction or script execution.
5. Inspect control layers. Review robots rules, authentication, CDN settings, firewall policies, bot challenges, redirects, and rate controls. A crawler can be allowed in one layer and denied by the next.
6. Repair the shared component. If a menu, card grid, pagination control, or related-content module causes the failure, fix that component instead of manually patching individual pages.
7. Review server logs after deployment. Look for requests to the repaired pages, the status codes returned, and whether crawling expands beyond the original entry points. Treat a user-agent label as a clue rather than proof of identity because request headers can be imitated.

Pay special attention to responses that are technically successful but operationally empty. A status of 200 is not sufficient if the body contains only an application shell, a consent wall, a challenge page, or a message that requires cookies. The content you want processed needs to be present in the response available to that crawler.

Redirects deserve the same scrutiny. A crawler that requests a public article but lands on a login screen, locale selector, or generic homepage did not receive the resource you intended. Follow the full redirect chain and assess the final body, not only the first response.

Measure technical coverage before AI visibility

After a repair, measure the outcome that the repair can actually influence. For discoverability work, that means crawler requests spreading from known pages to newly exposed URLs. For access work, it means intended responses replacing blocks, challenges, or empty shells.

Use a simple before-and-after record for the affected page set:

• How many intended URLs were reachable through HTML href links?
• How many received crawler requests during the observation period?
• Which response codes did those requests receive?
• Did the returned bodies contain the primary content?
• Did crawler activity expand from hubs into deeper pages after the change?
• Which crawler, template, and access layer were involved?

Keep this technical evidence separate from downstream visibility metrics. Crawl coverage can improve without an immediate citation or referral because selection is a later decision. That does not make the repair unsuccessful; it means you have removed one prerequisite and can now evaluate content quality, entity clarity, authority, and relevance without an access failure distorting the result.

Likewise, do not generalize one crawler’s behavior to every AI system. Rendering capability, supported controls, crawl frequency, and discovery routes can differ. Record results by named crawler rather than combining all automated traffic into a single AI-bot total.

Key takeaways

• A page cannot be selected or cited until a system can discover and retrieve it.
• A browser-visible navigation element is not necessarily a crawlable link. Important destinations should appear as ordinary anchors with href attributes in the delivered HTML.
• JavaScript can enhance navigation, but it should not be the sole source of important URLs.
• An llms.txt file is optional guidance, not a substitute for internal links and not a universal crawler-control mechanism.
• Test raw, unauthenticated responses through every relevant layer: origin server, CDN, firewall, bot protection, redirects, and application rendering.
• Use server logs to verify requests and responses, but do not mistake crawling for ranking, citation, or traffic.

Choose one high-value content path this week and follow it from entry page to deepest URL using only the original HTML and public responses. If the path breaks, repair the shared link or access layer first. That gives every later AI optimization effort a technically reachable foundation.

References

• genmark.ai Blog – Common Crawl Read 584,107 llms.txt Files. 68% Were Template Output.
• genmark.ai Blog – Six AI Crawlers Found Zero of 293 Pages