I noticed that Google updated its canonicalization troubleshooting guide to clarify how long it may take for fixes to appear in Google Search results. According to the revised guidance, Google might keep pages in a duplicate cluster for up to two weeks after content issues have been fixed.
What changed. I found a new section at the top of the guide that explains the expected timeline for canonicalization fixes. Google now makes it clear that the process can take up to two weeks.
I also saw additional technical details about clustering. Google explains that pages need to be sufficiently similar before its systems can group them into a duplicate cluster and select one version as the canonical page.
Google’s updated canonicalization guidance sets expectations for SEOs: fixed pages may remain in a duplicate cluster for up to two weeks, while clearer content differences can speed reevaluation.
Here is the section Google added:
Why I care. This clarification gives me a more realistic timeline when monitoring canonicalization fixes. Once Google has processed an update, I know I may need to wait the full two weeks before deciding whether the change worked.
As AI reshapes search, every glowing path to discovery carries commercial value—turning SEO investment into a conversation about pipeline, risk, and customer acquisition costs.
That waiting period can help me avoid making unnecessary page changes while Google is still consolidating duplicate URLs and evaluating the appropriate canonical version.
Let me be blunt: SEO advice can sound completely made up to people who do not live in search every day.
When I say things like “change this canonical,”“don’t block that resource,” or “we need this content exposed in the rendered HTML,” I understand why someone outside SEO might hear it and wonder whether I am inventing rules on the spot.
That is one reason SEO still gets treated like black magic inside many organizations.
I have been pushing the idea of “un-nerding SEO” for years, but this is about something very practical: I use Google’s own documentation to earn approval, build trust, and help SEO work get prioritized.
Not because Google tells us everything. Not because every sentence in its documentation should be treated as gospel. I use it because documented evidence is much harder to dismiss than personal opinion.
When I need buy-in, the strongest argument is rarely “trust me.”
It is usually something closer to: “Google has already documented how this should be approached.”
The buy-in problem is usually not the recommendation itself
In my experience, most SEO recommendations do not die because they are wrong. They die because they are competing with everything else happening inside the business.
Dev sprints, product timelines, CMS limitations, legal concerns, brand standards, executive assumptions, and the classic “we’ve always done it this way” all have a seat at the table. SEO is rarely the only priority in the room, even when the recommendation is technically correct.
That is why I do not rely on “best practice says” or “from an SEO perspective” when I am trying to move work forward. Those phrases sound optional, especially to teams already balancing risk, deadlines, and competing requests.
But “Google has official documentation that supports this recommendation” lands differently.
It may not automatically win the argument, and it definitely does not mean the work will be prioritized tomorrow. But it changes the conversation from “the SEO person said so” to “we have official Google documentation explaining why this matters.”
Google documentation is not gospel
I know the objection already: “Are we really pretending Google tells us the full truth about how search works?”
Absolutely not.
Google’s documentation is not the complete truth of search. It has omissions. It simplifies complex systems. Sometimes it explains how Google wants site owners to behave, not every technical factor that influences organic visibility.
Google also writes for a broad audience, which means nuance gets smoothed out, edge cases get skipped, and the answer can be technically true without being the entire story.
So no, I am not treating every Google statement as if it were carved into stone and carried down from Mountain View.
But that does not make the documentation useless.
It makes it a starting point. A receipt. An official reference point.
It moves the discussion away from “I think this matters” and toward “Google has explicitly documented why this matters.” That distinction matters when I am asking someone else to approve and prioritize the work.
Documentation is especially useful with developers
This is where Google documentation often earns its keep the fastest. SEOs need developers, and I have learned that the quickest way to lose developer support is to treat every recommendation like a command instead of a requirement that needs to be implemented thoughtfully.
And yes, just in case it ever works, I still wish I could run this:
google.exe /disable-ai-overviews /please
Bummer. No dice.
Developers are not wrong just because they disagree with an SEO recommendation. Most of the time, they are optimizing for completely valid priorities: performance, code quality, technical debt, security, and avoiding the kind of production mistake that can take a whole site down.
But sometimes developers are wrong about how Google discovers, crawls, renders, indexes, or interprets content.
And telling a developer “you’re wrong” is a great way to make sure my ticket never sees the light of day.
This is where documentation helps. It removes some of the subjectivity and shifts the discussion toward how to implement the requirement inside the existing technical environment.
The point is that I now have an external source of truth to discuss. That is a much better conversation than two teams arguing from preference.
Documentation is also a client management tool
For client-facing SEO work, documentation helps me separate serious recommendations from “trust me, bro, I have a contact at Google” consulting.
Rows of illuminated data cabinets and paper files stretch into the distance, capturing the pressure on marketers to turn fragmented customer data into a smarter performance engine.
That matters even more when a client has been burned by bad SEO advice before.
Instead of saying, “We need to change this because it’s better for SEO,” I can frame the recommendation with evidence.
“Here’s what Google documents. Here’s where your current setup conflicts with that. Here’s the risk. Here’s the recommendation. Here is the estimated reward.”
That framing builds trust because it shows the recommendation is not relying on blind faith.
It also makes the SEO look less like a magician and more like an interpreter.
That is how I see the real role of SEO: translating Google’s documented needs into business and technical decisions that a team can actually act on.
Less black magic, more receipts
SEO has a reputation problem, and some of it is earned.
Too much SEO work is still explained with vague phrases and shaky confidence. I hear people say things like “Google likes this” or “this needs to exist for the bots” when the stronger version is: “Google documents this behavior here, and here is how it applies to our situation.”
That does not mean documentation alone creates buy-in.
Dropping a Google link into a ticket or Slack thread is not a strategy. I still have to translate what it means, explain the risk, connect it to business outcomes, and help the team understand why the recommendation deserves attention.
Google documentation will never replace experience, testing, or judgment. It will not tell me everything, and I should not treat it like the final answer to every SEO debate.
But it can make SEO easier to defend, easier to prioritize, and much harder for leaders to dismiss.
The best SEOs are not just the ones who know what to recommend. They are the ones who can prove why the recommendation deserves to be taken seriously.
Less black magic. More receipts. More results.
Google documentation may not be the whole truth, but I would rather show up to a buy-in conversation with official references than with “my buddy from Google told me.” Suuuure they did.
This post first appeared on the author’s website and is republished here with permission.
After nearly 30 years at Microsoft, I am seeing one of Bing’s most influential search leaders close a remarkable chapter. Fabrice Canel announced that he is retiring from Microsoft, writing on LinkedIn, “I am retiring from Microsoft, effective today July 1st.” He also reflected, “Today marks nearly 30 years with Microsoft. Thirty years…”
When I think about Fabrice Canel’s impact, I think first about the foundation of Microsoft Bing Search. He was responsible for indexing at Bing, including crawling, URL discovery, content selection, and content processing. Those areas are core to how search engines understand the web, and Fabrice helped shape them at massive scale.
He was also the person behind the IndexNow initiative, and he played a major role in creating and powering Bing Webmaster Tools. For anyone working in SEO, publishing, or technical search, those contributions matter because they helped make discovery, indexing, and webmaster communication faster and more practical.
I have watched Fabrice contribute far beyond product work. He has spoken at countless industry events, including SMX, and has written extensively about how search works, how sites can perform better in Bing, and how search is evolving with generative AI. He helped run one of the world’s most important search engines, while also giving the SEO community tools, education, and direct insight.
In his retirement message, Fabrice addressed fellow Microsoftees, engineers, attorneys, marketers, webmasters, publishers, SEO champions, product leaders, journalists, people across search and AI, and even friends at Google. His note was warm, personal, and full of gratitude for the people who shaped his Microsoft journey.
He described his three decades at Microsoft as a wonderful adventure, from solving real business problems with IndexNow to helping webmasters and publishers thrive in the constantly changing world of SEO and AI. He thanked colleagues, partners, publishers, and the people he trained and mentored, saying they are ready to carry the mission forward.
Fabrice also shared that, after many conversations with family and friends, he decided to take advantage of Microsoft’s Voluntary Retirement Program. His message ended with the same sense of warmth and storybook style that many in the industry have come to associate with him: gratitude for Microsoft, confidence in the Bing team’s future, and a final wish that everyone stay curious, keep innovating, and make content easier to find.
Why do I care so much about this? Because Fabrice has been a true friend to the search industry. His work will live on through the products, systems, and initiatives he helped create, and his willingness to share knowledge has made a lasting difference for SEOs, publishers, developers, and search professionals.
I know Fabrice has trained a team to continue the work, and I believe Bing remains in good hands. Still, I would be lying if I said I am not sad to see him retire. It has been an honor to work with him and learn from him over the years, and his legacy at Microsoft Bing will be felt for a long time.
When I work on technical SEO, I know the right changes can dramatically improve how search engines crawl, understand, and evaluate a website.
I also know that the recommendations with the biggest upside usually carry the biggest implementation risk. URL changes, canonical updates, robots.txt edits, internal linking improvements, and site migrations can all strengthen organic performance, but one mistake can damage crawling, indexing, and search visibility.
That is why I do not treat technical SEO as a simple list of fixes. I treat it as a process: evaluate the impact, weigh the effort and risk, align the right teams, and test everything before and after launch.
From audit to implementation to prioritization
For me, the work is not finished when the SEO audit is delivered.
Prioritization is where the real judgment begins. I look at how severe the issue is, what outcome I expect, how many pages are affected, how much development effort is required, and what could go wrong if the change is implemented poorly.
The recommendations with the greatest potential impact often need buy-in from developers, content teams, product owners, and stakeholders because they require more resources and carry more risk. A clear recommendation, a practical test plan, and early alignment make implementation much easier to move forward.
Understanding the issue and potential outcome
I do not assume every technical SEO issue found in an audit needs immediate action. Before I prioritize a recommendation, I validate it manually and consider the broader context of the site, including priority sections, platform limitations, and business goals.
For example, missing meta descriptions on low-priority pages or title tags that fall outside recommended lengths may appear in an audit because they are easy for tools to measure, not because they will meaningfully affect performance.
Crawling tools and automated reports are valuable because they help me find issues at scale. But they do not always tell me whether an issue matters to the business.
A warning may point to a real problem, an intentional setup, a platform constraint, or something with little to no measurable impact. I need that context before I decide what deserves attention.
Evaluating impact, risk, and effort
Once I validate an issue, I decide how to address it and whether it is worth recommending for implementation.
When I am prioritizing technical SEO recommendations for a development queue, I consider the number of affected pages, the expected outcome, the resources required, and the potential risks.
Updating a few title tags may be low risk. Changing URL structures or modifying robots.txt directives can affect thousands of pages and influence crawling, indexing, and discoverability.
By understanding both the upside and the downside, I can make better decisions, allocate resources more responsibly, and plan changes in a way that reduces risk while still pursuing meaningful gains.
High-impact technical changes that require extra caution
The following technical SEO initiatives can meaningfully affect site performance. I do not avoid them because they are risky. I approach them carefully because their implications, benefits, and failure points need to be understood before implementation.
1. URL updates and changes
I often recommend URL updates when a site needs a clearer folder structure, content consolidation, rebrand support, or stronger information architecture.
For example, a business may move service pages from the root domain into a subfolder so the content is easier to organize and the site is easier to navigate.
URL changes can provide real benefits, but I need to make sure those benefits outweigh the risks and that a proper redirect strategy is ready before anything goes live.
Search engines treat a changed URL as a new URL, so redirects are essential for preserving rankings, traffic, backlinks, and other signals tied to the original page. Missing redirects, bad mappings, redirect chains, outdated internal links, and stale XML sitemaps can all hurt crawling, indexing, and discoverability.
Before I move forward with URL changes, I create a redirect mapping plan. Ideally, I validate and test redirects in a development environment before launch, then check them again after launch and update the XML sitemap.
I also include internal link updates and performance monitoring in the launch plan. Careful planning helps preserve existing SEO equity while supporting the larger goals of the site.
2. Canonical updates
Canonical tags help search engines understand which version of a page should be treated as the preferred version when duplicate or similar content exists. I use them to consolidate ranking signals, avoid internal competition, improve crawl efficiency, and clarify which URLs should be prioritized for indexing.
For example, an ecommerce site may use canonical tags to consolidate parameter-based URLs or faceted navigation pages to a primary product or category page. But if a canonical tag is applied to the wrong template, it could unintentionally tell search engines to consolidate an entire group of important pages elsewhere.
Canonical updates may look simple, but mistakes can be difficult to spot once they are deployed across a site. I take time to review canonical targets and validate the implementation so I do not send conflicting signals that cause important pages to lose visibility or fall out of the index.
3. Robots.txt file changes
The robots.txt file controls how search engines and other crawlers access content on a website. I usually recommend robots.txt changes to improve crawl efficiency, prevent low-value content from being crawled, or limit access to specific site sections.
For example, I may recommend blocking filtered URLs, internal search results, or other pages that consume unnecessary crawl resources. When implemented correctly, these updates help focus crawl activity on more important content.
The risk comes from rules that are too broad, misplaced, or copied from the wrong environment. A single directive can block important sections of a site from being crawled. Accidentally deploying a staging robots.txt file to production can also disrupt how crawlers access live content.
Because robots.txt changes can affect large parts of a site, I test rules carefully, review the proposed changes against the intended URL patterns, and verify the implementation after launch. Even a small robots.txt edit can have sitewide consequences.
4. Internal linking changes
Internal linking helps search engines discover content, supports priority pages, connects related topics, and guides users through a website. My recommendations may include updating navigation, adding contextual links, consolidating content hubs, or improving pathways to key pages.
As websites evolve, internal linking often needs cleanup. Removing important links, creating orphaned pages, linking to staging environments, or accidentally pointing users and crawlers to non-public URLs can all hurt discovery. Large navigation updates can also change how easily search engines reach important content.
That is why I always look closely at scope. A navigation update may touch thousands of pages, making it far riskier than adding a few contextual links to a small group of priority pages.
5. Site migrations
At some point, every SEO team deals with a site migration. It may happen because of a rebrand, a domain change, a redesign, or a move to a new CMS. When planned well, migrations can improve user experience, support long-term SEO performance, and benefit the business.
They are also inherently risky because they often combine several technical SEO changes at once. Redirects, URL restructures, canonical tags, indexing directives, content updates, and internal linking changes may all happen during the same launch. With that many moving parts, even a small oversight can affect crawling, indexing, and visibility.
Even a well-planned migration can run into problems if changes are not documented, tested, reviewed, and validated throughout the process. I rely on pre-launch QA, post-launch testing, and ongoing monitoring to catch issues before they have a lasting effect on performance.
Working across teams to ensure success
Technical SEO updates often require multiple teams to work together. I may need input from content teams, in-house developers, external agencies, product managers, and analytics teams before a change is ready to launch.
Clear communication is essential. I make recommendations straightforward, build testing and QA into the process, and define success criteria before launch. I also want a plan for quickly identifying and resolving issues if something goes wrong.
Communicating recommendations effectively
Whether I am discussing a recommendation directly with developers or documenting it in a structured ticket, I make sure the issue is clearly defined, examples are included, and the required changes are easy to understand.
Clear documentation helps me set expectations, explain scope, identify affected URLs, and define the expected outcome. It also gives teams a place to ask questions, raise concerns, and flag limitations before implementation begins.
Testing in development environments
Whenever a site change is made, I want it tested thoroughly before launch. A development environment gives me a place to validate the implementation, ask questions, and provide feedback while there is still time to adjust the work.
Post-launch testing and monitoring
Sometimes a change that works perfectly in development behaves differently after launch.
That is why I am ready to validate the implementation as soon as changes go live. Post-launch checks help me identify issues quickly, begin troubleshooting immediately, and monitor the impact before small problems become larger ones.
Balancing opportunity and risk
Most technical SEO recommendations are designed to improve crawling, indexing, or site architecture. When I implement them correctly, they can significantly improve how search engines access, understand, and evaluate a website.
But implementation usually depends on multiple teams working toward the same goal. As a recommendation moves from audit to production, misunderstandings, assumptions, and overlooked details can create unintended consequences.
That is why I see technical SEO as more than finding opportunities. I need to understand the issue, evaluate the potential impact, weigh the development effort, and manage the risk of implementation.
No technical SEO change is completely risk-free. But with thoughtful planning, clear communication, thorough testing, and ongoing monitoring, I can catch issues earlier, reduce their impact, and roll out high-impact changes with the caution they deserve.
When I ran a crawl on my website, the report flagged hundreds of technical issues, all marked as high priority by my chosen tool. Sketching out a plan based on best practices, I felt the dread of impending communication with my developers.
But here’s the twist: Not all those ‘critical errors’ are really significant. I could spend weeks fixing high-priority technical issues and still not see a meaningful rise in traffic or conversions.
Some fixes seem urgent yet irrelevant, like a 404 error buried deep in the site architecture. It probably doesn’t deserve all the fuss.
Conversely, a minor issue in internal linking on high-value category pages might be holding millions of potential revenue back.
The real challenge in technical SEO isn’t in the fixes themselves but in understanding that not all issues hold the same weight. The myth that every fix is equally important persists. They simply aren’t.
Understanding the shift from issue-based to impact-based SEO is crucial for growth. Fixing everything isn’t the goal; fixing what truly moves the needle is.
Technical SEO tools are invaluable yet often create unnecessary anxiety. Crawl reports and health dashboards with flashing red flags often give the impression that every issue must be addressed immediately.
Yet, labeling something as a ‘critical issue’ due to a best practice violation doesn’t necessarily mean it impacts organic performance.
Time is often lost confusing technical correctness with search impact.
A site doesn’t need to be technically perfect to perform well in search engines. Equally, having an excellent CWV score doesn’t guarantee success if the wrong problems are prioritized. Some issues are cosmetic, some matter only at scale, and some relate to outdated best practices.
For me, successful technical SEO should focus on outcomes, not scores from various tools.
I often ask myself: Do this issue impact crawlability or indexing? Does it affect key sections of my site, like top-performing pages? Is there tangible evidence that it’s suppressing traffic or rankings? These questions help me prioritize effectively.
Equipped with the answers, I use a prioritization matrix to strategize effectively.
Some high-effort, low-impact fixes often drain my time without real benefits, such as fixing 404 errors that don’t affect user journeys or chasing minor Core Web Vitals changes that don’t benefit key pages.
By focusing on strategic internal linking or fixing canonical issues, I achieve low-effort, high-impact wins that significantly enhance discoverability and performance.
I’ve realized that the context of every site differs. Factors like business models and site architecture change the impact of specific SEO practices.
There’s no universal checklist for SEO priorities. What matters is understanding the impact of a fix on my site’s unique structure and content, and how it generates value from search.
A crawl report might show thousands of errors, but not all spell opportunity. At times, a single fix like a canonical correction or rendering issue overshadows everything else.
The essence of real SEO expertise is distinguishing between insignificant noise and impactful changes.
When my website’s traffic suddenly vanished, it felt like my online presence had evaporated overnight. Google had stopped indexing my pages, and I was desperate to reverse the decline caused by a botched migration.
This is my journey through a challenging case study of a multinational media organization that lost 90% of its traffic after a domain migration. By addressing the underestimated issue of soft 404 errors, we managed to liberate traffic potential across 13 country-specific domains.
While the events unfolded between 2021 and 2023, the lessons I’ve learned are timeless, and they apply to anyone facing indexing hurdles today.
The Sudden Traffic Plunge
In January 2022, the Brazilian version of a cryptocurrency news website completed a domain migration. Shockingly, instead of a minor drop, traffic plummeted drastically. A comparison between December 2021 and December 2022 showed a decline of approximately 90% year-over-year in both sessions and pageviews.
Before the migration, our old domain (xx.com.br) enjoyed between 15,000 to 25,000 clicks per day. After shifting to a new subdomain structure (br.xx.com), traffic fell to a sustained rate of just 2,000 to 4,000 clicks daily, and it stayed that way for over a year.
The migration occurred alongside three major Google algorithm updates in June 2021: a core update, a spam update, and a page experience update. The Brazilian site, however, showed no recovery even after facing temporary volatility due to these updates.
More Than Just Redirects: The Migration Dilemma
Generally, traffic recovery following domain migrations occurs within weeks or months as Google recrawls the site. Here, we observed no such recovery.
The crux of the issue was that Google continued crawling the old domain long after the migration. This split Google’s crawl budget, not consolidating on the new domain as expected, severely hindering our SEO efforts.
In mid-August 2022, after fixing the migration problems with the help of my SEO and IT teams, I noticed a slight positive change—a peak of 12 clicks and 37 impressions on August 29. This gave me a sign that Google was beginning to recognize the new domain appropriately.
Utilizing Facebook Prophet forecasting on our pre-migration data, we estimated that without migration issues, the Brazilian site could have exceeded 2 million monthly clicks by early 2022. Instead, the numbers were far less impactful.
Deciphering the Indexing Bottleneck
Resolving the migration unveiled a deeper issue affecting all 13 country domains: a massive backlog in indexing.
Google processes pages through four stages: Crawl, Render, Index, and Rank. For the Brazilian site, while crawling new articles took just about 2 minutes—acceptable for news—indexing took 24 hours. This delay was disastrous for timely cryptocurrency news.
The Magnitude of Migration Chaos: 513,000 Unindexed Pages
Google Search Console data in January 2023 highlighted severe indexing challenges across all domains, with Brazil alone having 513,369 pages categorized as ‘Crawled – currently not indexed’.
The ‘Crawled – currently not indexed’ status was troubling. These pages weren’t indexed because Google deemed them low quality or duplicate—yet potentially valuable content was left out of the index.
Upon investigation, I discovered that automatically generated thin-content pages, like currency converter URLs (e.g., “usd-to-thor”), were eating up the crawl budget, deprioritizing the domain.
Dealing With Soft 404 Explosions
Addressing the migration alone wasn’t enough, as a surge of soft 404 errors also demanded attention. These errors occur when pages return a success status (200), but lack meaningful content, mystifying search engines and squandering crawl budgets.
Soft 404s were proliferating across domains, including the main site and several international versions, complicating our SEO efforts further.
In France, this accumulation of soft 404 errors caused Google’s crawl requests to drop drastically, illustrating the pressing need to fix these issues.
Tackling the Crawl Budget Crisis
Understanding crawl budget is crucial. Excessively crawling ineffective pages depletes Google’s ability to find and index valuable content, particularly harmful for news sites needing prompt indexing.
By early 2023, our technical SEO was draining crawl resources, leading to slower indexing of fresh content and lost online visibility.
Implementing a Systematic SEO Fix
On January 31, 2023, I initiated an all-encompassing SEO strategy to target three priorities at once: Resolving soft 404s, optimizing the crawl budget, and refining Core Web Vitals, though the latter took a backseat to immediate indexing concerns.
Key actions included proper HTTP status code implementations for non-existing pages, optimizing URL structures, and improving canonicalization.
After the Fixes: Impressive Traffic Rebounds
The results were measurable just weeks later. In Brazil, ‘Crawled – currently not indexed’ pages fell by 57%, soft 404 errors reduced by 69%, and traffic began trending upward in early 2023.
International Recovery Highlights
In Germany, indexed pages surged, driving total daily clicks notably higher. Similarly potent results emerged across Poland and Spain.
Key Insights from My SEO Journey
I learned that handling indexing issues trumps almost every other SEO concern. No matter the quality of your content and backlinks, if your pages aren’t being indexed, your visibility won’t improve.
Moreover, ignoring soft 404s can quietly erode your site’s crawl budget, which silently undermines your SEO efforts until it becomes glaringly apparent in lost traffic.
Finally, detailed verification during domain migrations and focusing SEO strategies on regional requirements can make all the difference between an underperforming and a thriving website.
When I first looked at my SEO data, everything seemed perfectly fine. All metrics from Google Search Console, traffic, and indexing were normal without any red flags. But then, I decided to dig deeper using Scrunch, our AI citation monitoring tool, to examine the platform presence for searchinfluence.com over the past 30 days.
Here’s what I found: Google AI Mode showed a presence of 37.8%, Copilot at 22.2%, Google Gemini at 16.3%, ChatGPT at 9.6%, and Perplexity at 7.8%. Alarmingly, both Claude and Meta AI were at 0.0%.
Two platforms had zero presence. Given that every crawler reads the same site, differences in content quality or topical authority couldn’t explain this discrepancy. The only factor that varied was crawler access.
To understand this further, I analyzed seven days of Cloudflare logs and discovered 29,099 bot requests, with 65.8% involving AI bots. The requests rate-limited with HTTP 429, or “too many requests,” were interestingly varied by bot user-agent.
Training crawlers that make bulk requests are throttled, while user-facing crawlers that mimic human pacing during live queries aren’t. For example, ClaudeBot made 20,583 crawl requests for each referral returned.
My assumption was that the 429 errors originated from Cloudflare, perhaps due to a web application firewall (WAF) or security plugin interference. I went down a rabbit hole investigating multiple layers. It was time-consuming and ultimately unnecessary.
The truth emerged when I performed a reproduction test using curl requests, revealing that the block was based on user-agent, not path or rate. The realization hit when I discovered the x-powered-by header: WP Engine hosted our site, and the block came from their platform infrastructure.
I then tested other AI bot UAs and crafted a fingerprint for each, discovering that the blocklist was outdated. While some bots were blocked, others like Common Crawl passed through unaffected.
In conclusion, while WP Engine’s firewall, documented on their support page, was intended as a security measure, it wasn’t transparent to customers. Identifying these blocks requires specific diagnostic steps, and the process taught me much about managed hosting’s hidden layers.
I’ve always been fascinated by how Google keeps improving its search capabilities. Recently, Gary Illyes from Google shared more about Googlebot’s operations, diving into its crawling ecosystem, fetching processes, and how it handles data.
Googlebot Reimagined. It’s intriguing to learn that Google uses multiple crawlers for diverse objectives. Referring to Googlebot as a singular entity might not capture this complexity anymore. You can find more details on the various crawlers and user agents here.
Understanding Limits. During a recent discussion, Google elaborated on its crawling limits. Gary Illyes provided these insights:
Googlebot fetches up to 2MB for any individual URL, except for PDFs.
This means it crawls only up to 2MB of a resource, encompassing the HTTP header.
For PDF files, the limit is notably higher at 64MB.
Image and video crawlers have varied threshold values, contingent on the product they serve.
By default, other crawlers have a 15MB limit, regardless of content type.
What exactly occurs when Google initiates crawling?
Partial Fetching: For HTML files exceeding 2MB, Googlebot will not dismiss the page. Instead, it halts the fetch exactly at the 2MB mark, including HTTP request headers.
Processing the Cutoff: The downloaded section is then forwarded to Google’s indexing systems and the Web Rendering Service (WRS) as if it were the entire file.
The Unseen Bytes: Any data beyond the 2MB cutoff won’t be fetched, rendered, or indexed.
Resource Handling: All referenced resources in the HTML, except media, fonts, and certain files, are fetched by WRS independently, with their own byte count not affecting the parent page’s size.
Rendering Bytes with Google. Once the crawler accesses these bytes, WRS takes over. It processes JavaScript and executes code like a modern browser to grasp the final visual and textual state of the page. This process doesn’t request images or videos but does respect the 2MB threshold for each resource.
Best Practices You might want to embrace these recommended practices:
Streamline Your HTML: Shift large CSS and JavaScript to external files. While the main HTML document is capped at 2MB, external scripts and stylesheets can be fetched separately, under their own constraints.
Prioritize Content: Position crucial elements like meta tags, <title>, <link>, canonicals, and vital structured data high in the HTML to ensure they’re not overlooked.
Monitor Server Logs: Keep track of server response times. If your server struggles to deliver data efficiently, our fetchers may slow down to avoid overloading, reducing crawl frequency.
Don’t Miss the Podcast! Google also released a podcast on this topic. Check it out:
When faced with limited development resources, it can feel impossible to address all technical SEO issues. I’ve found that focusing on architecture, indexing, and performance can drive significant gains.
Let me share my insights on tackling technical SEO effectively, even when resources are stretched thin. Technical issues often stall progress, but they remain a top priority for many SEOs and are crucial for Google’s rankings.
One major hurdle I encounter in-house is the limited resources available for implementing website changes. In fact, up to 67% of respondents in Aira’s State of Technical SEO Report cite non-SEO development tasks as a key impediment to making necessary changes.
This shortfall in resources potentially costs businesses millions in revenue every year, according to seoClarity. Therefore, I believe it’s essential to focus on the tasks that deliver the highest impact.
Let’s dive into where you should focus first by employing prioritization techniques. Aira’s report suggests focusing on quick wins, KPI impacts, user impact, best practices, and staying ahead of industry changes.
For massive sites, it can be hard to know where to start. I suggest working with small groups of keywords or specific product areas, addressing barriers to ranking, ensuring key pages are indexed, and improving or eliminating low-quality pages.
Conducting a technical SEO audit helps pinpoint the exact issues needing attention and provides a prioritized list of tasks. SEO tools and resources like SEO prioritization techniques can guide you in focusing on impactful areas.
Now, let’s discuss key foundational fixes, starting from site architecture, an essential part of a smooth-running SEO program. Proper site structure ensures effective crawling, indexing, and user experience, setting your site up for search success.
By organizing your site around search patterns, you align content and navigation with the keywords and queries users are likely to use, thus enhancing the search and user experience.
With effective internal linking, you reinforce your site’s topical authority, guiding users to priority pages and turning your site structure into a ranking powerhouse.
However, in large enterprises, a full site architecture overhaul can be daunting. I recommend focusing on specific tasks that can be implemented with minimal resistance, such as strengthening internal links and consolidating overlapping topics.
Focusing on critical issues like crawling and indexing inefficiencies can dramatically improve search visibility. By addressing high-priority indexing issues first, you ensure search engines can accurately reflect your most valuable content.
It’s crucial to eliminate signal dilution, which can occur from conflicting URLs or poorly managed canonical tags. Reducing crawl waste ensures that search engines focus their efforts on high-value pages, not unnecessary ones.
Regarding website performance, a site that’s difficult to use can squander hard-earned organic traffic. Sites like Yelp and Pinterest have shown how investments in site performance can significantly increase user engagement and revenue.
Start by fixing any backend bottlenecks, as they impact everything from speed to user experience. Optimizing JavaScript and rendering processes can further enhance site interactivity and stability.
Improving your website’s perceived speed by prioritizing essential elements for users can make a notable difference in performance scores. Focus on preload strategies, inlining critical CSS, and reserving layout space for dynamic content.
For mobile user experience, it’s vital that your site not only looks good but performs optimally on mobile devices. With the majority of traffic coming from mobile, ensuring your site’s content and navigation are mobile-friendly is critical.
Think about how to optimize for mobile-first indexing, easy navigation, and emerging behaviors like voice search to retain and convert mobile users effectively.
By building momentum through strategic technical SEO wins, focusing on site structure, crawlability, and user experience, I know you’ll set your SEO program on a path to success, even with limited resources.
Where does my content lose its impact on AI systems? The answer lies in the five crucial stages: discovery, selection, crawling, rendering, and indexing.
The journey of my content doesn’t stop at creation. The DSCRI-ARGDW pipeline maps the ten gates my content must pass through before AI systems recommend it. Among these, the initial five infrastructure gates are discovery, selection, crawling, rendering, and indexing.
This infrastructure phase is critical—it determines whether my content is even visible to AI systems. As each stage passes, confidence in my content can degrade, leading to missed opportunities downstream.
If the content can’t be rendered, it might still get indexed, albeit with incorrect information. Thus, every competitive gate that follows relies on the surviving information.
When the core content is compromised, no competitive strategy can save it. This intricate process has been simplified into a checklist labeled “crawl and index,” but each step is an opportunity for optimization.
Even if you’re a seasoned technical SEO expert, don’t skip this. You might be missing out on crucial improvements that could ensure your content reaches indexing with maximum confidence.
The infrastructure gates are sequential—each gate’s success determines the next, and failure at any point can halt the entire process.
Starting with discovery ensures focus on the earliest failures. Tackling this first is more cost-effective than addressing later stages prematurely.
Discovery, selection, and crawling are well-known gates where content undergoes assessment, and understanding them is crucial for effective optimization.
Discovery is a signal-based process driven by XML sitemaps, IndexNow, and internal linking. Unfortunately, content that lacks entity association becomes an orphan in this system, waiting longer to be processed.
The process of selection is often ignored despite being a key determinant of the crawl budget. Less is more, a lesson from Microsoft Bing’s Fabrice Canel, highlights the importance of focusing on quality over quantity of pages.
Crawling, while vital, has become commonplace due to advancements in server response optimizations. However, rendering fidelity continues to be a significant challenge where much of the core content could be lost.
JavaScript can pose a challenge in this stage. Not all systems invest in executing it, leading to potential loss of vital content for bots.
To bypass JavaScript issues, consider alternatives like server-side rendering or new pathways through WebMCP, Markdown for Agents, or Cloudflare’s markup.
The conversion fidelity stage transforms the content once it passes rendering, but here it might face new challenges in preserving the integrity of information.
The indexing stage could fail if the system can’t determine which parts of a page are essential, making proper semantic markup crucial.
As I navigate these stages, from an absolute to a competitive test, structured data emerges as a powerful tool but only when used correctly.
Skipping stages such as rendering and maximizing confidence before competition gives my content a significant edge. Employ methods like WebMCP or IndexNow to innovate past existing stages.
In conclusion, paying attention to these infrastructure gates helps me preserve confidence in my content and leverage structured data effectively. This ensures that my competitive strategy in SEO starts on a strong foundation, prepared to face the ARGDW phases to come.
