Skip to content

Features
Pricing
Changelog
Status

Category: Technical optimization

Essential Checks for a Seamless Website Migration
I’ve learned that website migrations often fail due to small oversights. That’s why I focus on reducing risks with thorough pre-launch, launch-day, and post-launch SEO checks.

Website migrations can notoriously go awry, even with the best planning. I’ve seen rankings slip, traffic drop, and tracking break. Surprisingly, it’s usually the small oversights rather than complex technical issues that cause these problems.

I approach website migrations with a staging process. The checks I perform during staging, on launch day, and in the few weeks following the launch are crucial. They often determine whether a migration stabilizes quickly or spirals into a long recovery project.

Before Launch: Catch Issues on Staging

I’ve found that most migration problems should be identified and resolved on the staging site. If issues make it to the live site, recovery tends to be slower and more uncertain. Here’s how I set myself up for success:

Keep the Staging Site Private (Even from Crawlers)

A common mistake I’ve encountered is making the staging site publicly indexable. Google crawling a staging environment can lead to duplicate content in search results, causing rankings to fluctuate and unfinished pages to be indexed.

I make it a point to block crawlers from the staging site or protect it with a password to ensure it stays invisible to search engines until the live launch.

It’s not just about the crawlers. I’ve seen ecommerce sites where customers found the staging site and tried to place orders, creating confusion and frustration internally.

Take Benchmarks

To help identify real issues rather than reacting to normal shifts, I always take a baseline. I record organic sessions, rankings, top landing pages, indexed pages, conversions, and site speed before moving to the new site.

Identify Priority Pages

For me, it’s crucial to focus on pages that drive traffic, revenue, or attract links. These need extra care during redirect mapping, content review, and testing, with special attention to internal links, redirects, and URL rules.

Review Templates and Content Continuity

Templates are the backbone of a website, controlling titles, headings, metadata, and more. If templates break, similar problems can spread across countless pages. Here’s what I check:
- Presence and accuracy of titles and headings.
- Canonical tags that use full URLs and point to live pages.
- Correctly transferred structured data.
- Intact copy, images, and internal links.
Launch Day: Verify Everything Works on the Live Site

On launch day, preparation meets reality. I join my SEO, developer, and design teams to make sure what worked on staging works on the live site as well. Even small oversights can immediately impact rankings, traffic, and user experience.

Test Redirects at Scale

It’s not enough to spot-check. Every mapped URL should redirect correctly, without chains or loops, as they can slow down crawling and delay signal consolidation.

Crawl the Live Site

Immediately after the site goes live, I run a full crawl and compare the results to the staging crawl to spot any differences. I’m on the lookout for broken links, redirected internal links, missing pages, and server errors.

Check Internal Links and Navigation

Menüs, breadcrumbs, and in-content links should directly point to live URLs. Allowing internal links to rely on redirects adds unnecessary load and risk.

After Launch: Monitor and Stabilize Performance

I know that even with the best planning, surprises can emerge once search engines and real users start interacting with the site. Small errors missed on staging can suddenly affect rankings or traffic.

Structured monitoring in the days and weeks post-launch is crucial. By catching issues early, I can ensure they don’t impact performance or user experience.

Inspired by this post on Search Engine Land.
March 24, 2026
Master Technical SEO: Prioritize for Maximum Impact

When faced with limited development resources, it can feel impossible to address all technical SEO issues. I’ve found that focusing on architecture, indexing, and performance can drive significant gains.

Let me share my insights on tackling technical SEO effectively, even when resources are stretched thin. Technical issues often stall progress, but they remain a top priority for many SEOs and are crucial for Google’s rankings.

One major hurdle I encounter in-house is the limited resources available for implementing website changes. In fact, up to 67% of respondents in Aira’s State of Technical SEO Report cite non-SEO development tasks as a key impediment to making necessary changes.

This shortfall in resources potentially costs businesses millions in revenue every year, according to seoClarity. Therefore, I believe it’s essential to focus on the tasks that deliver the highest impact.

Let’s dive into where you should focus first by employing prioritization techniques. Aira’s report suggests focusing on quick wins, KPI impacts, user impact, best practices, and staying ahead of industry changes.

For massive sites, it can be hard to know where to start. I suggest working with small groups of keywords or specific product areas, addressing barriers to ranking, ensuring key pages are indexed, and improving or eliminating low-quality pages.

Conducting a technical SEO audit helps pinpoint the exact issues needing attention and provides a prioritized list of tasks. SEO tools and resources like SEO prioritization techniques can guide you in focusing on impactful areas.

Now, let’s discuss key foundational fixes, starting from site architecture, an essential part of a smooth-running SEO program. Proper site structure ensures effective crawling, indexing, and user experience, setting your site up for search success.

By organizing your site around search patterns, you align content and navigation with the keywords and queries users are likely to use, thus enhancing the search and user experience.

With effective internal linking, you reinforce your site’s topical authority, guiding users to priority pages and turning your site structure into a ranking powerhouse.

However, in large enterprises, a full site architecture overhaul can be daunting. I recommend focusing on specific tasks that can be implemented with minimal resistance, such as strengthening internal links and consolidating overlapping topics.

Focusing on critical issues like crawling and indexing inefficiencies can dramatically improve search visibility. By addressing high-priority indexing issues first, you ensure search engines can accurately reflect your most valuable content.

It’s crucial to eliminate signal dilution, which can occur from conflicting URLs or poorly managed canonical tags. Reducing crawl waste ensures that search engines focus their efforts on high-value pages, not unnecessary ones.

Regarding website performance, a site that’s difficult to use can squander hard-earned organic traffic. Sites like Yelp and Pinterest have shown how investments in site performance can significantly increase user engagement and revenue.

Start by fixing any backend bottlenecks, as they impact everything from speed to user experience. Optimizing JavaScript and rendering processes can further enhance site interactivity and stability.

Improving your website’s perceived speed by prioritizing essential elements for users can make a notable difference in performance scores. Focus on preload strategies, inlining critical CSS, and reserving layout space for dynamic content.

For mobile user experience, it’s vital that your site not only looks good but performs optimally on mobile devices. With the majority of traffic coming from mobile, ensuring your site’s content and navigation are mobile-friendly is critical.

Think about how to optimize for mobile-first indexing, easy navigation, and emerging behaviors like voice search to retain and convert mobile users effectively.

By building momentum through strategic technical SEO wins, focusing on site structure, crawlability, and user experience, I know you’ll set your SEO program on a path to success, even with limited resources.

Inspired by this post on Search Engine Land.

March 18, 2026
Unlocking SEO Success: Mastering the Five Key Infrastructure Gates

Where does my content lose its impact on AI systems? The answer lies in the five crucial stages: discovery, selection, crawling, rendering, and indexing.

The journey of my content doesn’t stop at creation. The DSCRI-ARGDW pipeline maps the ten gates my content must pass through before AI systems recommend it. Among these, the initial five infrastructure gates are discovery, selection, crawling, rendering, and indexing.

This infrastructure phase is critical—it determines whether my content is even visible to AI systems. As each stage passes, confidence in my content can degrade, leading to missed opportunities downstream.

If the content can’t be rendered, it might still get indexed, albeit with incorrect information. Thus, every competitive gate that follows relies on the surviving information.

When the core content is compromised, no competitive strategy can save it. This intricate process has been simplified into a checklist labeled “crawl and index,” but each step is an opportunity for optimization.

Even if you’re a seasoned technical SEO expert, don’t skip this. You might be missing out on crucial improvements that could ensure your content reaches indexing with maximum confidence.

The infrastructure gates are sequential—each gate’s success determines the next, and failure at any point can halt the entire process.

Starting with discovery ensures focus on the earliest failures. Tackling this first is more cost-effective than addressing later stages prematurely.

Discovery, selection, and crawling are well-known gates where content undergoes assessment, and understanding them is crucial for effective optimization.

Discovery is a signal-based process driven by XML sitemaps, IndexNow, and internal linking. Unfortunately, content that lacks entity association becomes an orphan in this system, waiting longer to be processed.

The process of selection is often ignored despite being a key determinant of the crawl budget. Less is more, a lesson from Microsoft Bing’s Fabrice Canel, highlights the importance of focusing on quality over quantity of pages.

Crawling, while vital, has become commonplace due to advancements in server response optimizations. However, rendering fidelity continues to be a significant challenge where much of the core content could be lost.

JavaScript can pose a challenge in this stage. Not all systems invest in executing it, leading to potential loss of vital content for bots.

To bypass JavaScript issues, consider alternatives like server-side rendering or new pathways through WebMCP, Markdown for Agents, or Cloudflare’s markup.

The conversion fidelity stage transforms the content once it passes rendering, but here it might face new challenges in preserving the integrity of information.

The indexing stage could fail if the system can’t determine which parts of a page are essential, making proper semantic markup crucial.

As I navigate these stages, from an absolute to a competitive test, structured data emerges as a powerful tool but only when used correctly.

Skipping stages such as rendering and maximizing confidence before competition gives my content a significant edge. Employ methods like WebMCP or IndexNow to innovate past existing stages.

In conclusion, paying attention to these infrastructure gates helps me preserve confidence in my content and leverage structured data effectively. This ensures that my competitive strategy in SEO starts on a strong foundation, prepared to face the ARGDW phases to come.

Inspired by this post on Search Engine Land.

March 10, 2026
Uncover the Impact of the DOM on SEO and Web Crawling

Have you ever wondered how the structure of your webpage affects its visibility on search engines? As someone who regularly dives deep into the technicalities of SEO, understanding the DOM (Document Object Model) is crucial for optimizing your site.

I’ve often encountered discussions about the DOM with developers, and maybe you’ve seen it referenced in tools like Google Search Console. But why does it matter so much for SEO? Let me walk you through its significance and how to optimize it.

In essence, the Document Object Model is the browser’s dynamic, in-memory representation of your webpage. It serves as a bridge that allows programs, notably JavaScript, to interact with your content.

The DOM is structured like a family tree:

The document: Acts as the root of this tree.

Elements: HTML tags such as <body> and <p> transform into branches or nodes.

Relationships: There are parent-child-sibling relationships among elements.

This hierarchy is key for the browser and search engines in understanding your content’s structure, helping them discern, for instance, which paragraph is associated with a given heading.

The exploration of the DOM doesn’t end there. Let’s look at how you can inspect it directly.

The DOM, a JavaScript object, can be viewed in a format akin to HTML using browser DevTools—just right-click on your page, select Inspect > Elements, and you’ll see the Elements panel.

In this panel, it’s easy to dive into the structure by:

Expanding and collapsing nodes to explore hierarchy,

Searching for elements using Ctrl+F (Cmd+F on Mac), and

Identifying JavaScript-added or -modified elements as they flash briefly on change.

However, do remember that this tool sometimes shows a different view from what Googlebot crawls. I’ll delve into this discrepancy a bit later.

Next, understanding how the DOM is built is essential. It starts with the browser converting the HTML file retrieved from a server line-by-line into tokens, which are then turned into nodes forming a tree structure.

This tree-building process allows browsers to create a hierarchical structure necessary for rendering the web page you see, which also includes building a CSS Object Model (CSSOM), but this is less crucial for SEO than the DOM.

JavaScript often runs during this DOM construction. On encountering a <script> tag without async or defer attributes, the browser pauses to execute the script before continuing. These scripts might modify the DOM by adding content or changing links, differing from the initial HTML code.

Let me illustrate this: Each click on a button dynamically adds a paragraph to the DOM, changing the page’s visible content.

The original HTML is just a starting blueprint; the final constructed DOM is what the browser utilizes. It can dynamically change based on JavaScript operations.

Why does the DOM matter for SEO? Modern search engines like Google render pages using headless browsers (Chromium). They evaluate the DOM, not just the initial HTML response.

Googlebot’s crawl process includes parsing HTML, executing JavaScript, and taking a DOM snapshot for indexing. However, remember:

Googlebot doesn’t interact with pages like humans—content triggered by user actions might go unnoticed.

Other crawlers might not render JavaScript, missing out on JavaScript-dependent content.

With AI agents harnessing DOM data for task execution, a well-structured and accessible DOM becomes ever more crucial.

Verifying what Google sees via Google Search Console’s URL inspection tool reveals the rendered HTML version indexed by Google, showcasing any issues.

Using this tool can alert you to discrepancies in what Google indexes versus what you expect, impacting your SEO efforts if overlooked.

For instances without console access, you can resort to Google’s Rich Results Test for similar page insights.

To ensure your webpages are crawled and indexed well, here are some best practices:

Make sure significant content loads in the DOM by default—Googlebot doesn’t interact beyond initial page loads.

Use proper <a> tags to ensure links are crawlable, avoiding JavaScript-based navigation that search engines don’t execute.

Maintain a clear semantic HTML structure. Search engines rely on tags like <header>, <article>, and <section> to understand content organization, unlike ambiguous <div> nesting.

Keep your DOM lean—under about 1,500 nodes—to avoid performance lags and enhance user experience.

In a digital landscape increasingly reliant on AI interactions and advanced crawling methods, understanding and optimizing the DOM is key to maintaining your site’s SEO competitiveness.

Inspired by this post on Search Engine Land.

March 5, 2026
Mastering Anthropic’s Claude Bots: Control and Blockade Guide

Have you ever wondered how all those Claude bots from Anthropic handle your site’s data? Well, I’ve delved into their latest update, which offers insights into their AI training, real-time queries, and what happens when you choose to block them.

Anthropic recently enhanced their crawler documentation, providing clarity on how Claude bots interact with websites and how you can regain control by blocking them.

Why should you care? If you’re like me and manage content, you’ll want to manage how AI systems utilize your work. Anthropic smartly divides bots into training crawlers, user-initiated fetches, and search indexers. Blocking just one won’t impact the others, so make informed choices based on visibility and training implications.

Let’s meet the robots: Anthropic employs three unique user agents. First up, ClaudeBot gathers public online content for training their AI models. Blocking it means your site’s content won’t be in future AI datasets.

Next, there’s Claude-User, which fetches pages when someone asks Claude a question necessitating site access. Block this bot and lose out on visibility in user-driven response queries.

Finally, Claude-SearchBot improves search results by indexing. If you decide to block it, it may affect your content’s visibility and accuracy in Claude-enhanced search responses.

Curious about blocking these bots? They comply with standard robots.txt directives, including “Disallow” and “Crawl-delay”. To block a bot site-wide, use:

User-agent: ClaudeBot
Disallow: /

Bear in mind, each bot and subdomain you wish to limit needs its own directive. Be cautious with IP blocking; these bots operate via public cloud IPs, which might interfere with robots.txt access, and IP details aren’t disclosed by Anthropic.

Explore Anthropic’s documentation here: Does Anthropic crawl data from the web, and how can site owners block the crawler?

Inspired by this post on Search Engine Land.

February 25, 2026
Cloudflare’s Markdown Feature: A Game Changer or a Cloaking Risk?

Yesterday, I stumbled upon some exciting news from Cloudflare. They’ve introduced a feature called Markdown for Agents, which provides machine-friendly versions of web content alongside the traditional pages we all see.

Cloudflare describes this update as a proactive measure in response to increasing AI crawler activities and agentic browsing.

When a client requests text/markdown, Cloudflare fetches the HTML from the origin server, converts it right at the edge, and then hands over a Markdown version.

Interestingly, the response includes a token estimate header, which helps developers like me manage context windows more effectively.

Early feedback highlighted not only the efficiency gains but also the potential implications of offering alternate representations of web content.

What’s happening. Being part of the 20% of the web that Cloudflare powers, I learned that Markdown for Agents utilizes standard HTTP content negotiation. If a client sends an Accept: text/markdown header, Cloudflare immediately converts the HTML response on-the-fly to Markdown format. The response, marked with Vary: accept, ensures caches store separate versions.

Cloudflare views this opt-in feature as a shift in content discovery and consumption, benefitting AI crawlers and agents with its structured text that requires less overhead.

They claim Markdown can reduce token usage by up to 80% compared to HTML, which is quite impressive!

Security concern. SEO consultant David McSweeney raised a concern, citing that Cloudflare’s Markdown for Agents feature might make AI cloaking incredibly simple because the Accept: text/markdown header tips off origin servers that the request is AI-related.

Regular requests deliver the usual content, but those for Markdown can trigger a unique HTML response that gets converted for AI consumption, McSweeney explained on LinkedIn.

The worry is that sites might inject hidden instructions, altered product data, or other machine-only content, creating a hidden “shadow web” for bots, unless the header is stripped before reaching the origin.

Google and Bing’s markdown smackdown. Here’s the kicker. Representatives from Google and Microsoft advised against creating separate markdown pages for large language models. Google’s John Mueller noted:

“Given that LLMs have always trained on and parsed normal web pages, it seems obvious they have no issues with HTML. Why serve a page that no end user sees? Plus, if they validate equivalence, why not stick to HTML?”

Microsoft’s Fabrice Canel added:

“Do you really want to double crawl load? We’ll check for similarity anyway. Non-user versions (like crawlable AJAX) are often neglected and broken. Human oversight fixes both user and bot views. Schemas help, and AI makes us even better at deciphering web pages. Less is more in SEO!”

Cloudflare’s feature doesn’t generate another URL but does create varied representations based on request headers.

The case against markdown. Technical SEO consultant Jono Alderson pointed out that once a machine-targeted representation exists, platforms must choose to trust it, verify it against the human version, or outright ignore it:

“Flattening a page to markdown doesn’t only remove clutter. It strips away judgment and context.”

“The instant you publish a machine-exclusive page representation, you craft a secondary candidate version of reality. Regardless of source promises or claims of identical content, a system now views two representations and must determine the true reflection of the page.”

Dig deeper. Why LLM-only pages aren’t the answer to AI search

Why we care. With Cloudflare’s advancements, AI ingestion might become more cost-effective and streamlined. But does serving distinct content to humans and crawlers verge on cloaking? Stay tuned…

Inspired by this post on Search Engine Land.

February 13, 2026
Boost Your Website’s AI Visibility: Overcome Crawling Hurdles

Have you ever wondered why your site isn’t getting the attention it deserves from AI crawlers? I know how frustrating it can be to feel overlooked in the digital world. Often, Cloudflare might be the culprit blocking access.
Let me guide you through diagnosing these issues, providing solutions, and optimizing your site for better LLM (Large Language Model) visibility. Together, we’ll ensure your site is primed for the AI-age and ready to capture its rightful place in search rankings.

Inspired by this post on HiGoodie Blog.

December 29, 2025
Mastering Canonicalization for SEO and GEO Success in 2026

Canonicalization and SEO: A Personal Guide for 2026

Canonicalization has always been pivotal in SEO, yet it’s surprisingly easy to overlook. In 2026, managing duplicate content and optimizing for generative engines is becoming essential. Let’s explore this together.

Canonicalization helps search engines pinpoint original content sources and prevent duplicate versions from competing. This is a must-know for large sites aiming to stay organized and small ones looking to avoid ranking dilution.

As 2026 approaches, canonicalization is gaining even greater traction with the rise of generative engine optimization (GEO), alongside traditional SEO. AI and tools like ChatGPT are reshaping content selection and attribution processes. Let’s dig into why this matters.

This guide will walk you through essential canonical tags, practical strategies for implementation, and advanced insights benefiting both SEO and GEO.

What is canonicalization?

Canonicalization, a cornerstone of technical SEO, allows you to specify the preferred version of a webpage when similar content exists across different URLs. Think of it as designating the primary source or ‘master copy.’

Using canonical tags effectively tells search engines which URL to index and rank, sidestepping confusion and focusing your site’s authority and ranking power on the right page.

Key terms

The crucial terms we’ll cover include canonical tag, self-referencing canonical, origin, target URL, and duplicate content. Grasping these will enhance your understanding as we delve deeper.

Why canonicalization matters for SEO and GEO

Canonicalization is crucial for boosting SEO and GEO performance. It enables search engines to consolidate sources and choose the authoritative page while generative systems respond to precise canonical signals. Let’s explore the essentials of a solid strategy.

How to implement a canonical tag

You may need a developer to implement canonical tags, but many CMS platforms have features to add self-referencing canonicals automatically. However, some situations require manual specification for certain page types.

Practical applications for canonicalization

Deploying self-referencing canonicals even on unique content is a best practice. It ensures indexing efficiency and prevents confusion. Technical nuances like www/non-www, HTTP/HTTPS variations, and URL parameters can present issues that canonical tags can address.

Let’s also look at cross-domain canonicalization, pagination strategy, and managing ecommerce complexities associated with product variations and faceted navigation, ensuring your implementation remains current with 2026 best practices.

The role of tools and monitoring

Monitoring canonicalization through Google Search Console, Screaming Frog, and similar tools is critical. Catching issues early prevents them from affecting rankings. Regular checks for canonical conflicts ensure your strategy’s success.

Canonicalization trends to watch

With search evolving rapidly, canonicalization is now integral not just for managing duplicates but as a foundational signal for both indexing and appearing in AI-generated answers. Keeping up with 2026 trends will ensure your strategy remains effective.

Takeaways on canonicalization

Mastering the fundamentals of canonicalization, maintaining URL hygiene, and tailoring strategies to specific site needs are crucial. Regular monitoring and adapting to ongoing changes, especially with AI’s impact, sustains your site’s health and authority.

Inspired by this post on Search Engine Land.

November 25, 2025

←Previous Page

1 2