CrushPress.AI

Tag: Search Optimization

  • Google’s Legal Battle: SerpApi Accused of Unlawful Data Scraping

    Google’s Legal Battle: SerpApi Accused of Unlawful Data Scraping

    Today, I came across an intriguing development where Google has initiated legal proceedings against SerpApi. This lawsuit revolves around allegations that SerpApi has been bypassing Google’s security systems to scrape and resell copyrighted content from search results.

    The Allegations: According to Google, SerpApi has:

    • Circumvented the security measures and standard crawling controls Google has in place.
    • Ignored directives from websites that specify content accessibility.
    • Employed techniques such as cloaking, rotating bot identities, and large bot networks to scrape vast amounts of content.
    • Appropriated licensed content from search features such as images and real-time data, subsequently selling it for profit.

    Google’s Stance: Describing SerpApi’s actions as “brazen” and “unlawful,” Google expressed concerns over how stealthy scrapers like SerpApi override crawling directives, stripping sites of their choices. Alarmingly, Google noted a significant increase in SerpApi’s activities over the last year.

    Quick Update: Interestingly, Google’s lawsuit mirrors similar legal action by Reddit, which also targeted SerpApi, Perplexity, Oxylabs, and AWMProxy. Reddit accused them of scraping content via Google Search results and concealing their identities to evade restrictions.

    • Reddit has licensing agreements with Google and OpenAI, suspecting other entities of attempting to bypass these deals.
    • They reportedly set a “trap” post, visible only to Google’s crawler, which eventually surfaced in Perplexity’s results as proof of scraping.
    • SerpApi denied these allegations, claiming their operations are lawful.

    SerpApi’s Previous Statements: In defense, SerpApi has maintained that “public search data should be accessible,” viewing its actions as protected by the First Amendment. They also warned that lawsuits like the one from Reddit could endanger the “free and open web.”

    Why It Matters to Me: Should Google triumph in this case, acquiring reliable SERP data might become increasingly challenging and costly. This could particularly impact teams reliant on services like SerpApi, as they navigate the complexities of understanding search results, performance metrics, and achieving success in an evolving digital landscape.

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot
  • Boosting AI Search Visibility: Avoiding Duplicate Content

    Boosting AI Search Visibility: Avoiding Duplicate Content

    Have you ever considered how duplicate content might be impacting your visibility in AI search results? Fabrice Canel and Krishna Madhavan from Microsoft recently discussed how duplicate content complicates AI search systems, reducing the chances of selecting the correct version for summarization.

    Much like traditional search engines, AI search platforms such as Bing and Google rely on consistent intent signals. When your content appears in duplicate forms, it can confuse these systems, making it difficult for them to interpret signals accurately.

    The Impact of Duplicate Content on AI Search. Here are key takeaways from the Bing blog about the impact of duplicate content:

    AI search utilizes traditional SEO signals while also adding layers to understand user intent.

    Repeated content across multiple pages weakens intent signals, complicating AI interpretation, and selection.

    If several pages contain similar content, AI cannot easily identify which aligns with user intent, reducing preferred page selection chances.

    Large Language Models (LLMs) cluster near-duplicate URLs, often selecting outdated versions if variations are minimal.

    Campaign pages and localized versions must differ meaningfully; identical content provides less matching signal.

    AI favors updates, but duplicates can slow the process of updating system information.

    The Challenge of Syndicated Content. Many might not realize syndicated content—articles republished on various sites—can also be problematic. Microsoft considers this duplicate content because identical articles across domains make it difficult for search engines and AI to identify the original source.

    Strategies to Minimize Duplicate Content. If you deal with syndicated content, ask partners to:

    • Use canonical tags directing to the original version on your site.
    • Rework content for uniqueness.
    • Noindex republished articles to prevent search engine indexing.

    Organizing Campaign Pages for Clarity. Microsoft warns that campaign pages with only minor changes can still be considered duplicates. To manage this:

    • Designate a primary campaign page for interaction.
    • Apply canonical tags to variations without unique intent.
    • Maintain separate pages for distinct intents like seasonal offers or local pricing.
    • Redirect outdated or redundant pages to consolidate content.

    Handling Localization Pages. Localization can also produce duplicate content if differences are minimal. Microsoft suggests:

    • Introduce meaningful local variations with examples, terminology, or regulations.
    • Avoid multiple same-language pages for identical purposes.
    • Use hreflang to define language and regional targeting accurately.

    Addressing Technical SEO Concerns. Technical issues can lead to URL duplication, often managed automatically by search engines. However, it’s best to prevent this by maintaining a single URL per content piece. Common problems include:

    • Utilize 301 redirects for URL consolidation.
    • Apply canonical tags when accessible versions are necessary.
    • Ensure consistent URL structures site-wide.
    • Restrict crawler access to staging or archived URLs.

    Why This Matters. While duplicate content is not a new issue in SEO, its importance extends into AI search. Familiarity with its impact on indexing and ranking can guide strategies for improved visibility.

    For more insights, visit the Bing Webmaster blog.

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot
  • Enhance SEO with AI: Aligning Search Intent Effectively

    Enhance SEO with AI: Aligning Search Intent Effectively

    When I think about improving my website’s visibility, AI comes to mind as a crucial tool. It serves as a second pair of eyes, helping me evaluate intent signals, compare top results, and refocus pages that aren’t performing well.

    Despite having well-written content, excellent layout, and robust backlinks, pages can still underperform in rankings. A frequent culprit is misaligned search intent, which can be more elusive than it seems.

    Focusing on content optimization and usability sometimes makes it easy to overlook or misjudge intent. This is where AI shines as a reviewing tool, effectively steering things back on course.

    Whether I’m working on a new page or revising an existing one, returning to the basics of search intent always sets me up for success.

    Starting with a simple AI prompt to outline likely search intents for a keyword offers a solid framework for content creation or optimization.

    This comprehensive list isn’t something I strive to cover completely on a single page. Instead, it highlights diverse user types, shifts in intent, and needs I might not have initially considered.

    By considering these factors, I aim to create a more useful, well-rounded page that genuinely satisfies user needs.

    Dig deeper: There are more than 4 types of search intent

    Getting the intent right can be challenging. AI tools help me understand what’s already successful by examining top-ranking pages and what they excel at.

    I utilize AI tools for a swift overview of a page’s primary intent. By evaluating this at scale, I can see if top-ranking pages meet the same intent.

    It’s crucial to assess the intent of my page with the same rigor, be it a fresh draft or a page I’m optimizing. If the primary intent aligns with what’s succeeding, it’s a strong starting point. If not, it provides clear direction for improvement.

    Again, consulting AI tools for improvement suggestions can yield valuable insights into refining intent. Key areas to focus on include:

    The language I use can either reinforce or contradict the intended message. For commercial intent, persuasive wording is necessary, while for informational pages, clear and descriptive language is preferred.

    The format of a page can also convey intent. For instance, in a sales page, details like product placement and accompanying information matter greatly. Similarly, guides need clear step-by-step labeling and possibly visual aids.

    Clearly defined calls to action are essential. They align the user’s actions with the page’s intent, enhancing both engagement and ranking potential. Unclear or generalized calls to action dilute this effect.

    Dig deeper: How to master user intent with SEO personas

    Listing accurate pricing, VAT elements, and currency signals is vital in conveying commercial intent. They guide users accurately at critical decision points.

    Availability of support is another crucial factor. I make sure that pre- or post-sale queries can be easily addressed by ensuring my contact details and support options are clearly visible.

    Trust signals, like product guarantees, return policies, and customer reviews, make a big difference in user decisions. Including these details serves to strengthen user trust.

    When clear comparisons are needed, laying out products side by side can assist users in their decision-making process, moving them closer to making a purchase.

    In my experience with working pages centered around user intent, I’ve seen that excess information can sometimes bloat a page.

    Previously, this depth might have worked, but now clarity and a focus on intent are what truly resonate.

    I’ve learned to reassess where content performs best within the user journey, often seeking AI’s guidance to refocus content structure wisely.

    For instance, if I notice my sales page for internal French doors isn’t performing, I consult AI, along with competitor analysis, to uncover key insights.

    Competitors might be focusing on selling first, while my page addresses user concerns, which means I need to reposition my content priorities.

    By reordering sales-driven content and addressing pain points concisely, I better align with user intent, letting supporting pages deal with detailed post-sale information.

    AI isn’t here to replace expertise but to guide my strategic intent, enhancing my understanding of user behavior for better conversion.

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot
  • Google Resolves Page Indexing Report Delay for Instant Insights

    Google Resolves Page Indexing Report Delay for Instant Insights

    I just discovered that Google Search Console has finally addressed the month-long delay in their page indexing report. It felt like a stroke of relief when I saw the report now displays data as recent as just a few days ago. This is the usual schedule for updates and I’m thrilled to have this crucial tool back on track.

    Another piece of good news: emails notifying about indexing issues have resumed, so we site owners can stay informed directly through our inboxes once again.

    Page Indexing Report. This indispensable tool helps me understand which pages on my site Google can find and index, along with highlighting any potential problems. It also allows me to submit any fixes and see if they’ve been successfully implemented. Previously, when the report was delayed, we couldn’t verify our fixes or check the indexing status of new and existing pages, which was quite frustrating.

    Fixed Issues. Here’s a screenshot of the updated report, showcasing a more current date—December 14th—instead of the long-stagnant November 21st:

    ```json { "alt": "Google Search Console page indexing report with internal issue notice and last update date of 12/14/25.", "caption": "A glimpse into Google Search Console showing a page indexing report struggling with internal data update issues, marked with a last update date of 12/14/25.", "description": "This image shows a screenshot of Google Search Console's page indexing report, indicating an internal issue preventing recent data updates. A red arrow points to the last update date, 12/14/25, underlining the delay in data refresh. The interface features a clear search bar, an 'export' button, and user account controls, emphasizing typical dashboard usability. Keywords: Google Search Console, indexing report, data update, internal issue." } ```

    Besides, Google has also resolved the delays in performance reports just yesterday! It’s a relief to know that most significant reports are now functioning smoothly, though it’s always wise to stay prepared for potential future hiccups.

    Why It Matters to Us. Over the past month, many of us struggled to provide up-to-date reporting to our SEO clients and stakeholders. With the recent fixes, we now have access to the latest data for both page indexing and performance reports.

    This means we can get back to speed with our reporting tasks, just in time before the holiday season sets in.

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot
  • AI Bots Boom, But Googlebot Still Reigns Supreme in 2025

    AI Bots Boom, But Googlebot Still Reigns Supreme in 2025

    In 2025, Googlebot once again led the charge in generating more web traffic than any other crawler, as revealed in a new report by Cloudflare. Google continued its tireless web crawling for both search indexing and AI training, proving its dominance over other search and AI bots.

    According to the numbers from Cloudflare, Googlebot was responsible for more than 25% of all Verified Bot traffic. In fact, Googlebot alone accounted for 4.5% of all HTML request traffic, which is more than all other AI bots combined at 4.2%.

    The surge in AI “user action” crawling, which is when bots simulate human behavior, saw an impressive 15x increase year over year. Despite the rise in AI crawlers, Googlebot still had a crawl volume that eclipsed every other AI bot, including those from OpenAI, Anthropic, and Meta.

    In the world of AI crawlers, they were the most frequently disallowed in robots.txt files. Moreover, Anthropic notably had the highest crawl-to-refer ratio among major AI and search platforms, crawling much more content than it returned as traffic. Early in the year, this ratio hit ~500,000:1, before settling between ~25,000:1 and ~100,000:1 after May, as compared to OpenAI’s peak at ~3,700:1 in March and Perplexity’s lowest among major platforms.

    Diverse search platforms exhibited different behaviors. Microsoft’s ratio oscillated between ~50:1 and ~70:1, with a notable weekly cycle. Google’s ratio climbed from just over ~3:1 to ~30:1 by April, dropped to ~3:1 by mid-July, then gradually increased again. Meanwhile, DuckDuckGo stayed below 1:1 until jumping to ~1.5:1 in mid-October.

    Despite these changes, Google maintained its monopoly in search, delivering almost 90% of search engine referral traffic. Bing, Yandex, Baidu, and DuckDuckGo completed the top five, but their shares were significantly smaller.

    Throughout the year, very little shift occurred; Google remained dominant as Yandex’s share dipped from 2.5% to 1.5%, and Baidu experienced a modest rise from 0.9% to 1.6%.

    I found the full report quite insightful. If you’re interested in exploring it yourself, you can check out The 2025 Cloudflare Radar Year in Review for comprehensive insights on AI, post-quantum advancements, and notable DDoS attack trends.

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot
  • Discover ‘Read More’ Links in Google Search Snippets

    Discover ‘Read More’ Links in Google Search Snippets

    Recently, I noticed that Google has introduced ‘read more’ links right in the search result snippets. When you click these links, they take you directly to a specific section of a webpage that is relevant to your query.

    While not every search snippet includes these links, I’ve found that many do, offering a deeper dive into the content right from the search results.

    What it looks like. You can check out a screenshot of how these links appear. It’s quite likely that you’ll encounter them if you perform a search now.

    I remember Google testing similar features back in July, and it seems they’ve finally rolled it out more broadly.

    ```json { "alt": "Google search results page showing 'see more' links highlighted with red arrows.", "caption": "Google's new search feature: enhanced 'see more' links direct users to detailed content sections.", "description": "This image displays a Google search results page with multiple entries showcasing the new 'see more' hyperlinks, highlighted by red arrows. These links are Google’s initiative to provide direct access to specific content sections, promising users a more efficient navigation experience. With terms like 'Google AI Mode' and 'Web Guide', the image reflects an evolving digital landscape where keyword relevancy is key." } ```

    Why we care. These ‘read more’ links introduce an extra clickable area in the snippets, potentially driving more traffic to websites, which is always a positive outcome.

    Ultimately, more clicks mean more engagement with our content, so I’m hopeful that this feature is here to stay.

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot
  • Win an All Access Pass: Help Shape SMX Advanced 2026!

    Win an All Access Pass: Help Shape SMX Advanced 2026!

    I’m excited about the opportunity to influence the future of search marketing events. You can help shape SMX Advanced 2026 by sharing your insights and preferences. The event is happening from June 3-5 at the Westin Boston Seaport, and we want to know what you’re eager to learn and who you’re interested to hear from.

    Reflecting on June’s event, it was thrilling to reunite in person for the first time since 2019 at SMX Advanced. It was more than just a conference; it felt like a global reunion for search marketers to connect, share ideas, and dive into cutting-edge insights.

    The world of search is ever-evolving, with swift changes in AI SEO, algorithm updates, and the delicate balance of AI with a human touch. Advanced, actionable education is more crucial than ever, and that’s where you come in.

    Help Shape SMX Advanced 2026

    Our aim for SMX Advanced 2026 is to make it the most relevant and exciting yet, but we need your expertise to get there. Your input is invaluable, and we’re inviting you to directly influence the 2026 curriculum.

    Completing our brief survey lets you help build a program that addresses the critical challenges and opportunities you’re facing. Share with us:

    • Which advanced topics will boost your professional growth.
    • The search changes and complexities that concern you the most.
    • Experts and innovators you’re excited to hear from.
    • Preferred session formats, whether deep-dive clinics, lightning talks, or interactive panels.

    Fill out the survey here.

    Be Entered to Win an All Access Pass

    As a token of our appreciation, everyone completing the survey gets a chance to enter an exclusive drawing.

    One lucky winner will receive an All Access pass to SMX Advanced 2026! Join us for this landmark event at the Westin Boston Seaport from June 3-5.

    Submit a Session Pitch

    Beyond influencing the agenda, we’re offering you the chance to submit a session pitch. If you’ve developed a groundbreaking strategy or have valuable insights, lead the conversation and showcase your expertise.

    Check out our guide to speaking at SMX for details on submitting your session idea. When you’re ready, create your profile and send us your pitch.

    I’m looking forward to your submissions and insights! If you have questions, feel free to reach out to me at kathy.bushman@semrush.com.

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot
  • Google Search Console Performance Reports: Delay Resolved

    Google Search Console Performance Reports: Delay Resolved

    Screenshot of Google Search Console

    After enduring weeks of delays, I’m relieved to see that the performance reports in Google Search Console are finally back to normal. These reports had been set back by over 50 hours, creating quite the challenge for anyone relying on timely data.

    Latest Status: If you access the search performance report now, you’ll notice the data delay is significantly reduced to just 2 to 6 hours, which is fairly typical. Compare this to the over 70-hour delays we experienced recently!

    This is exactly what I observe:

    The delays started a few weeks back and it took about three weeks for Google to resolve the issue, clearing the backlog in the process.

    ```json { "alt": "Interface features including icons for help, account settings, notifications, apps, and export option.", "caption": "Navigate with ease using intuitive icons for help, account settings, notifications, and more. Export your data seamlessly with up-to-date features.", "description": "This image showcases a user interface section with various icons, including a question mark for help, a user icon for account settings, a bell for notifications, and a grid for app access. An export option is available, highlighting recent updates with a timestamp indicating the last update was 2.5 hours ago. These features are part of a streamlined access panel designed for user convenience." } ```

    Page Indexing Update: Unfortunately, the page indexing report delay we mentioned earlier is still not resolved. It’s frustrating to note that this delay has stretched to nearly a month, with Google acknowledging the issue in a notice:

    “Due to internal issues, this report has not been updated to reflect recent data”

    Why This Matters: If you depend on Search Console for analytics and reporting to clients, these delays can be incredibly frustrating. While it’s a relief that performance reports are up-to-date, the ongoing delay in the indexing report can still disrupt reporting processes.

    Meanwhile, Google has rolled out several exciting new features recently, including:

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot
  • Unlocking Social Media: Your New Search Engine Guide

    Unlocking Social Media: Your New Search Engine Guide

    I’ve come to realize that social platforms are revolutionizing how we discover information. TikTok, Reddit, YouTube, and AI engines are now essential in shaping the way we search.

    Understanding this shift, I’ve explored how brands can enhance their visibility by optimizing for social-driven Answer Engine Optimization (AEO). Let’s dive into these exciting developments!

    Inspired by this post on HiGoodie Blog.

    crushpress.ai community screenshot
  • Celebrating 19 Years of Search Insights and Innovation

    Celebrating 19 Years of Search Insights and Innovation

    Search Engine Land turns 19

    Today, I am thrilled to share that Search Engine Land is celebrating its 19th anniversary!

    Nineteen years is an incredible milestone. For almost two decades, we have been diving deep into the ever-evolving world of search engines, always striving to make sense of the changes and challenges Google and the search industry present.

    This year, 2025, has been one of the most transformative since our launch in 2006. The rapid pace of change has been exhilarating.

    Through it all, our mission remains steadfast: to provide clear news, insightful analysis, and practical guidance to help you navigate the world of search.

    Before we look to the future, I want to express my heartfelt thanks for your support and reflect on the past year with you.

    Thank you for reading

    Sincerely, thank you for being with us.

    Every day, we focus on you: what you need to know, what really matters, and what changes will impact your work today or your strategy months down the line.

    Our goals include:

    • Focusing on meaningful stories, not filler.
    • Delivering news clearly and quickly.
    • Providing essential context and expertise.
    • Being a dependable resource in a fast-changing industry.
    • Helping you anticipate where search is heading, even when it’s unclear.

    If you haven’t yet, I encourage you to subscribe to our daily newsletter for a curated summary of all things search, helping you stay updated without feeling overwhelmed.

    Thank you to the Search Engine Land team

    Our team’s passion is what has driven our success for almost two decades.

    Though small, our team accomplishes significant and impactful work because we are mission-driven and dedicated to search.

    I extend my greatest thanks to:

    • Barry Schwartz. With 22 years of experience, Barry’s passion for search ensures complex topics become understandable. He is indispensable.
    • Anu Adegbola. Focusing on paid media, Anu offers clarity amidst constant changes with her insightful writing.
    • Angel Niñofranco. Angel plays a crucial role in our SME articles through his coordination and editorial oversight.
    • Kathy Bushman. Kathy’s behind-the-scenes expertise ensures SMX events are seamless and valuable.

    And to the entire team at Third Door Media within Semrush, whether or not your name appears here, your contributions are invaluable.

    Top highlights from the past year

    Despite the uncertainties of this year, Search Engine Land thrived, thanks to the trust of our community.

    SMX Advanced returned in person for the first time in 6 years

    This was arguably the highlight of the year. SMX Advanced’s return in person after six years was electrifying.

    With attendance surpassing expectations, the sessions were dynamic, and conversations felt like reunions for the search marketing community. It was clear that we all missed these face-to-face exchanges about AI, Google’s updates, and more.

    We learned again that when great minds gather, extraordinary things happen. We eagerly await our next gathering in Boston, June 3-5.

    Defining industry coverage of AI Overviews and the new era of search

    This year, more than ever, transformed the search landscape. We’ve provided the clarity and reporting needed in this evolving environment.

    Our readers rely on us for insights during times of change, and we take pride in shaping the industry’s future understanding of search.

    Subject Matter Expert (SME) program growth

    This year saw a surge of new and returning readers turning to us for insight into SEO and PPC shifts, from AI to SERP experiments.

    Our growth owes much to our fantastic contributors, and I extend my gratitude for their impactful work.

    Looking ahead: What’s next for Search Engine Land

    As we embrace our 19th year, our resolution is steadfast: to offer unparalleled coverage of search-related topics.

    This year, you can anticipate:

    • Continued breaking news on SEO, PPC, AI, and more.
    • In-depth analysis, guides, and contextual explainers on industry evolution.
    • SMX events tailored around the nuances of AI-driven search.
    • Enhanced expert viewpoints, data, and market clarity.

    Mark your calendars for:

    • SMX Advanced: June 3-5
    • SMX Next: Nov. 18-19

    We have much in store for you, with the aim of equipping you with the insights necessary for your best work.

    A brief look back to where it all began

    Launched on Dec. 11, 2006, Search Engine Land began with a vision of search as a vast community. A place of exploration, connection, and evolution. Over these years, it’s grown beyond our expectations.

    The mission remains the same:

    Search Engine Land is your destination to remain informed, educated, and connected within the world of modern search engines.

    Thank you for 19 incredible years

    From everyone here at Search Engine Land and Semrush, thank you for your readership, engagement, and passion for the evolving world of search.

    Here’s to a promising rest of 2025 and a remarkable 2026.

    Inspired by this post on Search Engine Land.

    crushpress.ai community screenshot