AI Search Crawlability: A Technical SEO Audit Framework

Your pages can perform well in Google and still be effectively missing from AI-generated answers. The problem is often not the writing. An AI crawler may be blocked, unable to discover links, or receiving an HTML shell that omits the content and structured data people see in a browser.

You can diagnose that problem without guessing about prompts or rewriting every page. Audit the route from robots.txt to the raw server response, then fix the first point where a retrieval bot loses access, discovery, or meaning.

Key takeaways

• Audit the initial HTML response, not just the rendered page in your browser. Critical links, text, headings, metadata, and JSON-LD should be present before JavaScript runs.
• Treat training crawlers, search or retrieval crawlers, and user-initiated browsing agents as separate policy decisions in robots.txt.
• Use server-side rendering, static generation, or a hybrid approach for anything an AI system must discover, understand, or cite.
• Use server logs to distinguish a crawlability failure from a selection failure. A page that was never requested has a different problem from a page that was fetched but not cited.

Crawlability has three gates, and robots.txt is only the first

A useful AI crawlability audit separates access, discovery, and extraction. Combining them into one pass-or-fail score hides the actual repair.

Passing one gate proves nothing about the next. An Allow rule cannot make a client-rendered product description appear in the response. An XML sitemap may expose a URL, but it cannot supply missing text or JSON-LD. A browser screenshot can show a complete page even when the crawler receives almost nothing.

Do not use Google rendering as a proxy for every other system. The crawler ecosystem includes agents with different jobs and different rendering behavior. A successful Google inspection therefore does not establish that an AI retrieval crawler can follow the same path or extract the same facts.

Set crawler access by purpose, not by the letters AI

AI platforms can operate more than one agent. One may crawl broadly for model training, another may retrieve information for search, and another may visit a URL in response to a user’s request. Blocking or allowing the entire family with an inherited rule can produce the opposite of your intended policy.

• Training-oriented access: Decide whether broad reuse of your content fits your publishing, licensing, and compliance policy. ClaudeBot is an example of a crawler identified for training.
• Search and retrieval access: If you want pages to be available for AI answers, inspect rules affecting agents such as Claude-SearchBot and OAI-SearchBot separately from training crawlers.
• User-initiated browsing: Agents such as Claude-User and ChatGPT-User may fetch a page when a person asks an assistant to visit or use it. Treat that behavior as its own access decision.

The names matter because a blanket policy is not a strategy. A publisher may reasonably block training while allowing retrieval. A regulated organization may choose a narrower policy. The technical requirement is that robots.txt express the decision you actually made rather than a rule inherited from an old template, security product, or previous agency.

1. Write down the intended outcome for training, retrieval, and user-initiated access before editing robots.txt.
2. Map every relevant user agent to one of those outcomes. Do not assume agents owned by the same company serve the same function.
3. Review specific user-agent groups as well as broad wildcard rules. Look for inherited blocks that catch retrieval agents unintentionally.
4. Test the resulting policy with the exact user-agent names, then fetch representative URLs to confirm that permitted agents receive normal responses.
5. Record who owns the policy and why. Otherwise, a future security or infrastructure change can silently reverse it.

Robots permission is necessary only when you want that agent to enter. It is not evidence that the agent can navigate the site or understand the response. Continue the audit even after the policy passes.

Put the discovery path and critical facts in the initial HTML

Client-side rendering creates the largest practical gap between what a person sees and what many AI crawlers receive. If the server sends an empty container and JavaScript later inserts navigation, body copy, product details, or schema, a crawler that does not execute that script encounters an incomplete page.

The risk is especially clear in internal navigation. During the first 27 days of a 41-day controlled crawl experiment, GPTBot and ClaudeBot each reached all 748 hierarchy pages exposed through hard-coded HTML and none of the hierarchy pages available only through JavaScript-injected links. Googlebot reached seven of 293 pages in the JavaScript group, or 2%, and 35 of 748 in the HTML group, or 5%.

Those percentages are not universal crawl-rate benchmarks. The experiment intentionally removed sitemaps, breadcrumbs, and other alternative discovery paths so that reaching a JavaScript-only child would demonstrate script execution. What it establishes is the mechanism: when the only route to a page is a link inserted after load, major AI crawlers may stop at the parent.

Different crawlers from the same organization are not interchangeable either. GoogleOther rendered enough JavaScript to reach 142 of the 293 JavaScript-group pages in that experiment, while Googlebot reached seven. Activity from a secondary agent does not prove that the crawler responsible for a particular search or retrieval function saw the same pages.

For every page you want an AI system to use, place these elements in the server-delivered response:

• Followable internal links: Category, topic, breadcrumb, related-content, pagination, and other important paths should use links with destinations present in the raw HTML. Keep XML sitemaps as an additional discovery route, not as a repair for invisible navigation.
• The primary answer: The page’s main text, headings, definitions, specifications, and other decision-critical facts should not depend on a client-side API call.
• Entity details: Names, authors, dates, prices, product attributes, and relationships should appear clearly where they are relevant to the page.
• Critical metadata: Do not rely on JavaScript to add information that a crawler needs to classify or interpret the page.
• Structured data: Put the applicable schema markup, including JSON-LD, in the initial HTML rather than injecting it after the application mounts.

Server-delivered structured data gives a no-JavaScript crawler explicit entity and relationship signals. It can reduce ambiguity around facts such as names, dates, authors, prices, and product attributes. It should describe information that is also supported by the page, not act as a hidden substitute for missing visible content.

You do not have to remove JavaScript from the site. Use static site generation for content that can be built in advance, server-side rendering for pages whose critical response must be assembled dynamically, or a hybrid model that renders essential content and navigation on the server while leaving filters, interactions, and enhancements to the client.

The implementation label is less important than the response. A framework can claim SSR while a particular component still fetches its text, links, or schema in the browser. Verify the actual HTML returned for the actual template.

Run an audit that ends in a template-level fix

Start with representative paths rather than a random list of URLs. Include a top-level hub, a child page, a deep page that depends on several internal clicks, and each commercially or editorially important template. The relationship between those pages is part of the test.

1. Fetch the raw response without executing JavaScript. Save the response body and relevant headers. In a browser, View Source is more useful for this check than the Elements panel, which normally reflects the post-JavaScript document.
2. Confirm basic access. Check the response status, redirect destination, robots rules, and any challenge or interstitial delivered to the chosen agent. A visually normal page in your own session does not prove that an unauthenticated crawler receives it.
3. Search the response for the primary information. Verify that the title, main heading, answer text, defining facts, authorship, dates, product information, and other page-specific content are present as text rather than empty component placeholders.
4. Trace the internal path. Starting at the hub, inspect the raw HTML for links to the next level. Repeat until you reach the deep sample. If the path disappears before JavaScript runs, you have found a discovery boundary.
5. Inspect JSON-LD in the response. Confirm that the intended schema type, entity properties, and relationships are present server-side and agree with the information a reader can see.
6. Compare raw and rendered output. Any critical element that exists only in the rendered document is a client-side dependency. Classify it as discovery, content, metadata, or structured data so the development request names the actual failure.
7. Review server logs. Group requests by user agent, path, response status, and time. Look for agents that reach hubs but consistently stop before child pages. Do not trust a user-agent string alone when identity matters; the controlled crawler experiment verified Googlebot and Bingbot through reverse DNS to exclude spoofed traffic.
8. Repair the shared template and retest the path. A server-rendering fix to a hub, navigation component, or JSON-LD component can restore access across many URLs. Confirm the new response before treating deployment as completion.

Interpret the failure pattern before changing content

• The agent never requests the URL: Check robots access and discovery first. The absence of a request is not evidence that the copy needs optimization.
• The agent requests hubs but not their children: Inspect the parent response for missing links. A repeated stop at the same directory level is a strong JavaScript-boundary signal when the child links are absent from raw HTML.
• The agent requests the page but receives a thin shell: Move the critical content and facts into SSR, SSG, or hybrid output. Changing schema alone will not supply the missing body content.
• The text is present but JSON-LD appears only after rendering: change how the markup is delivered. Server-render it and verify it in the response body.
• Training is allowed while retrieval is blocked: revisit the robots policy if AI search visibility is the goal. The configuration does not match that objective.
• The page is fetched with complete HTML but is not cited: crawlability has probably passed for that request. Retrieval, relevance, factual clarity, and citation selection are separate stages, so do not keep treating every absence as a rendering bug.

Begin with one high-value hub and its deepest important child. Make sure an intended retrieval agent can access both URLs and that the raw responses contain the links, main content, factual details, and JSON-LD needed to interpret them. Once that path passes, apply the repair at the template level and verify the result in your logs before commissioning another round of content rewrites.

References

• Search Engine Land – JavaScript links can make your pages invisible to AI search
• Search Engine Land – Technical SEO for AI search: 4 fundamentals that matter