I recently discovered a new help document from Google that explains how their web crawlers operate. This document aims to offer basic educational information about crawling, highlighting key resources available to site owners.
There are currently nine essential insights listed in the document, and they’re pretty enlightening!
Frequent crawling is a good sign! It indicates that your site’s pages contain fresh or highly relevant content that attracts attention. Google specifically mentions, “If we’re crawling your site a lot, it’s an indication your pages have fresh or highly relevant content that people want to find, and that our systems are recognizing that demand. Online shopping is a great example: we crawl ecommerce sites often so that our results will display retailers’ most up-to-date prices, promotions, and inventory status.”
What’s included in the guide? Here’s a quick overview, though I’d definitely recommend diving into the document for a detailed read. It’s not new information, but it serves as a beneficial refresher:
- What is crawling? In short, crawling is how Google “sees” the web.
- Google uses numerous crawlers, each tasked with different jobs.
- Repeat crawls help provide the freshest search results by catching the latest updates.
- Frequent crawling remains a positive indicator!
- With the increased complexity of pages over time, Google’s crawling has evolved.
- Crawling is automatically optimized.
- Google doesn’t access paywall or subscription content without consent.
- Site owners have control over what gets crawled and how.
- Respect for robots.txt and other instructions is a standard for Google’s crawlers.
Why does this matter? The art of crawling is a cornerstone of SEO, essential for being visible in Google Search and other platforms. This new help document can serve as a guide to enhance the crawlability of your site.
Inspired by this post on Search Engine Land.
Leave a Reply