When my website’s traffic suddenly vanished, it felt like my online presence had evaporated overnight. Google had stopped indexing my pages, and I was desperate to reverse the decline caused by a botched migration.
This is my journey through a challenging case study of a multinational media organization that lost 90% of its traffic after a domain migration. By addressing the underestimated issue of soft 404 errors, we managed to liberate traffic potential across 13 country-specific domains.
While the events unfolded between 2021 and 2023, the lessons I’ve learned are timeless, and they apply to anyone facing indexing hurdles today.
The Sudden Traffic Plunge
In January 2022, the Brazilian version of a cryptocurrency news website completed a domain migration. Shockingly, instead of a minor drop, traffic plummeted drastically. A comparison between December 2021 and December 2022 showed a decline of approximately 90% year-over-year in both sessions and pageviews.
Before the migration, our old domain (xx.com.br) enjoyed between 15,000 to 25,000 clicks per day. After shifting to a new subdomain structure (br.xx.com), traffic fell to a sustained rate of just 2,000 to 4,000 clicks daily, and it stayed that way for over a year.
The migration occurred alongside three major Google algorithm updates in June 2021: a core update, a spam update, and a page experience update. The Brazilian site, however, showed no recovery even after facing temporary volatility due to these updates.
More Than Just Redirects: The Migration Dilemma
Generally, traffic recovery following domain migrations occurs within weeks or months as Google recrawls the site. Here, we observed no such recovery.
The crux of the issue was that Google continued crawling the old domain long after the migration. This split Google’s crawl budget, not consolidating on the new domain as expected, severely hindering our SEO efforts.
In mid-August 2022, after fixing the migration problems with the help of my SEO and IT teams, I noticed a slight positive change—a peak of 12 clicks and 37 impressions on August 29. This gave me a sign that Google was beginning to recognize the new domain appropriately.
Utilizing Facebook Prophet forecasting on our pre-migration data, we estimated that without migration issues, the Brazilian site could have exceeded 2 million monthly clicks by early 2022. Instead, the numbers were far less impactful.
Deciphering the Indexing Bottleneck
Resolving the migration unveiled a deeper issue affecting all 13 country domains: a massive backlog in indexing.
Google processes pages through four stages: Crawl, Render, Index, and Rank. For the Brazilian site, while crawling new articles took just about 2 minutes—acceptable for news—indexing took 24 hours. This delay was disastrous for timely cryptocurrency news.
The Magnitude of Migration Chaos: 513,000 Unindexed Pages
Google Search Console data in January 2023 highlighted severe indexing challenges across all domains, with Brazil alone having 513,369 pages categorized as ‘Crawled – currently not indexed’.
The ‘Crawled – currently not indexed’ status was troubling. These pages weren’t indexed because Google deemed them low quality or duplicate—yet potentially valuable content was left out of the index.
Upon investigation, I discovered that automatically generated thin-content pages, like currency converter URLs (e.g., “usd-to-thor”), were eating up the crawl budget, deprioritizing the domain.
Dealing With Soft 404 Explosions
Addressing the migration alone wasn’t enough, as a surge of soft 404 errors also demanded attention. These errors occur when pages return a success status (200), but lack meaningful content, mystifying search engines and squandering crawl budgets.
Soft 404s were proliferating across domains, including the main site and several international versions, complicating our SEO efforts further.
In France, this accumulation of soft 404 errors caused Google’s crawl requests to drop drastically, illustrating the pressing need to fix these issues.
Tackling the Crawl Budget Crisis
Understanding crawl budget is crucial. Excessively crawling ineffective pages depletes Google’s ability to find and index valuable content, particularly harmful for news sites needing prompt indexing.
By early 2023, our technical SEO was draining crawl resources, leading to slower indexing of fresh content and lost online visibility.
Implementing a Systematic SEO Fix
On January 31, 2023, I initiated an all-encompassing SEO strategy to target three priorities at once: Resolving soft 404s, optimizing the crawl budget, and refining Core Web Vitals, though the latter took a backseat to immediate indexing concerns.
Key actions included proper HTTP status code implementations for non-existing pages, optimizing URL structures, and improving canonicalization.
After the Fixes: Impressive Traffic Rebounds
The results were measurable just weeks later. In Brazil, ‘Crawled – currently not indexed’ pages fell by 57%, soft 404 errors reduced by 69%, and traffic began trending upward in early 2023.
International Recovery Highlights
In Germany, indexed pages surged, driving total daily clicks notably higher. Similarly potent results emerged across Poland and Spain.
Key Insights from My SEO Journey
I learned that handling indexing issues trumps almost every other SEO concern. No matter the quality of your content and backlinks, if your pages aren’t being indexed, your visibility won’t improve.
Moreover, ignoring soft 404s can quietly erode your site’s crawl budget, which silently undermines your SEO efforts until it becomes glaringly apparent in lost traffic.
Finally, detailed verification during domain migrations and focusing SEO strategies on regional requirements can make all the difference between an underperforming and a thriving website.
Inspired by this post on Search Engine Land.
