Managing my website’s URLs efficiently is crucial to prevent crawlers from slowing it down. If you’re like me, you want your site to load fast, ensuring both visitors and search engines have a seamless experience.
Just the other day, I listened to Google’s latest insights on their year-end report for 2025. It was fascinating to hear Gary Illyes discuss on the Search Off the Record podcast about the major crawling challenges Google faces, like faceted navigation and action parameters, which make up a whopping 75% of the issues.
What’s the issue? Well, I’ve learned that crawling problems can seriously impact site performance, potentially making it unusable or inaccessible. Crawlers can sometimes get stuck in an infinite loop on a site, wreaking havoc on server performance.
According to Gary, once a set of URLs is discovered, the crawler has to check a significant portion to determine its quality. By the time this is done, the damage is done—your site slows down dramatically.
The Biggest Crawling Challenges Here’s what caught my attention as the major issues from the report:
- 50% relate to faceted navigation. These are very common in e-commerce sites where endless filtering options exist for products based on size, color, price, etc.
- 25% pertain to action parameters. These come from URL parameters that trigger actions instead of significantly changing page content.
- 10% involve irrelevant parameters like session IDs or UTMs.
- 5% are due to plugins or widgets that cause confusion by creating problematic URLs.
- 2% encapsulate other “weird stuff”, which includes strange issues like double-encoded URLs.
Why this matters to me is simple. A well-structured URL strategy keeps my server healthy, ensures quick page loads, and prevents search engines from misunderstanding which URLs should be indexed as canonical.
The Podcast: Here’s where you can listen to the discussion yourself:
Inspired by this post on Search Engine Land.
Leave a Reply