As someone deeply invested in improving my SEO processes, I’ve discovered an innovative way to transform my workflows using AI-powered tools that adapt to my unique methods.
By leveraging platforms like ChatGPT and Google’s Gemini, I can get standard on-page SEO reviews. However, these initial responses often feel generic and devoid of specific context related to my business needs.
This generic nature of AI is both its limitation and its potential opportunity. While out-of-the-box AI provides broad solutions, it lacks the personalization that comes from my own business insights.
Fortunately, tools like GPTs, Gems, and Claude Projects allow me to embed my SEO process into custom assistants, making the complex seem straightforward without needing complex coding skills.
I’ve also learned that large language models predict responses from a vast array of internet data, often resulting in average opinions rather than tailored advice for my business specifics.
In SEO, these broad opinions typically revolve around general content improvements and link building, which might not address the unique challenges I face.
What I needed was a tool that factored in my business’s unique landscape, including customer needs and competitive environment. That’s where the personalization of AI tools comes into play.
Contextualizing inputs to AI tools transforms them into powerful assistants that enhance my specific workflow, making it less about generic data and more about strategic insights.
The process of creating a customized AI tool is more about narrating my workflows rather than needing a deep technical background. Tools like GPTs and Gems have become essential as I package my expertise into reusable, intelligent assistants.
Among the various AI platforms, I find GPTs, Gems, and Claude Projects especially user-friendly for most of my SEO tasks. These platforms are intuitive, allowing even non-developers like me to transform repetitive tasks into automated, efficient processes.
However, generic SEO tools, despite their widespread use, don’t pay attention to my company’s unique strategic priorities, unlike the AI applications I’ve tailored to fit my specific needs.
Moreover, crafting personalized AI apps not only aids in SEO but also transforms how I manage and execute marketing strategies, encompassing tasks like keyword research and content strategy more effectively.
My takeaway is that the true value lies not in AI itself but in the expertise I embed into it. My hard-earned industry skills are the real product, and AI simply empowers me to scale my efforts more efficiently.
It’s been enlightening to see how enhancing my AI tools with my knowledge improves productivity, ultimately strengthening my business impact. This process of encoding my SEO knowledge into AI-propelled systems is groundbreaking and transformative.
AI search is reshaping the marketing landscape faster than anything I’ve seen before.
During my time at Zero Click NY, I witnessed how significantly AI search has evolved over the last six months and identified emerging features that might define its future.
Among all the discussions, these seven trends were the most compelling.
From the emergence of marketing engineers, to the way Claude and ChatGPT differ in results, and Claude’s rapid ascent in the business world over the past year, here are the key insights I gathered.
1. Every AI relies on different content
According to Profound data, only 8% of citations are shared between ChatGPT and Claude. This means 92% of the sources that ChatGPT refers to would not be recognized by Claude for the same inquiry. Essentially, a brand may have high visibility in one AI and not exist in another.
Moreover, each AI favors different types of content.
ChatGPT frequently indexes community content: Reddit, Quora, and forums make up around 16% of its citations.
In contrast, Claude cites listicles 36% and opinion content 13.2% of the time, compared to ChatGPT’s ~20% and 7.2%, respectively.
The disparity also applies to traditional search. A significant 64% of websites Claude cites appear in Google’s top 50 for equivalent queries, whereas it’s only 37% with ChatGPT.
Takeaway: It’s vital to inform stakeholders that AI visibility differs between LLMs, and strategic prioritization is necessary to reach your audience.
Track your visibility by engine because effective strategies in one platform may not translate to another. UGC helps drive ChatGPT while listicles boost presence on Claude.
2. Claude is quietly winning B2B — so sequence your optimization by audience
Claude may appear insubstantial in AI traffic-share charts, but it’s a different story when it comes to enterprise usage.
Web traffic doesn’t tell the whole tale. Anthropic derives about 85% of its revenue from enterprise and API usage, not visible in consumer data.
A critical chart from Ramp’s AI Index reveals the true penetration of Anthropic in the business sector. A year ago, only a small number of businesses used Anthropic. Now, it’s at 34.4%, surpassing OpenAI at 32.3%.
This insight led me to reconsider: if more business users are engaging with Claude and consumers are on ChatGPT, shouldn’t our optimization priorities reflect audience preferences?
Should B2B entities focus on Claude first, while B2C aim for ChatGPT visibility?
Currently, few distinguish between ChatGPT, Gemini, or Claude usage. This distinction is bound to grow.
3. ChatGPT ads are here, and this is what we’re seeing
The game has changed: competitors are securing visibility through ChatGPT ads. These ads are now live and available for self-serve directly within the chat interface.
Recent weeks also saw the debut of GPT 5.5, citation chips morphing into clickable links (leading to a 60% spike in referral traffic overnight), and Google integrating AI Mode into its main search functionality.
This wasn’t incidental. The hyperlinks are crucial for an ads business. Analyzing over 100,000 ad placements highlighted three essential revelations.
ChatGPT Ads match on topic
Ads align with topic similarity, not intent. Only 14% of real user prompts express commercial intent, yet 20% show ads, even if the prompt involves a math problem.
Embedding analysis indicates that ad titles and descriptions significantly influence which conversations you appear in, transforming them into tactical targeting tools.
Paying for ads
We have entered a “pay-to-play” era. Approximately one-fifth of ad placements appear when a direct competitor is mentioned, but only 8% of organic references belong to the associated brand.
Competitors are twice as likely to advertise around your brand’s organic mentions than you are.
For instance, Startup CRM Adia is targeting prompts mentioning Salesforce, with Salesforce responding by showing paid ads 40% of the time, defending their position even when organically mentioned.
Ad inventory is scarce and expensive
Currently, ChatGPT presents about one ad per conversation, with the median exchange spanning three turns. Only 30% of eligible users ever see ads, and CPMs/CPCs are about four times Meta’s rates.
Expect future changes: additional ad slots per reply, ads woven deeper into conversations, and engineered suggestions to prolong interactions, ultimately increasing inventory.
The insight: Understanding both organic AEO and paid defense strategies is essential. Monitoring your brand’s organic citations without tracking who advertises against them offers a partial view.
4. Claude is the most directly optimizable AI right now
Claude sources web content directly from Brave searches, not merely influenced by it, as discussed in the presentation I attended.
In recent testing by Profound, 79.2% of Claude’s citations were directly aligned with Brave’s top 10 search results for equivalent queries.
Reshuffling is minimal; no other AI model trusts its search provider so extensively.
This transparency makes Claude the most straightforward AI to optimize for: a visible index, checkable rankings, and, as we’ll explore next, predictable retrieval.
If I’ve convinced you of the importance of Claude for B2B, here’s your approach: identify where you stand on Brave for key prompts and use that as your roadmap for Claude visibility.
This level of transparency won’t last forever. Take advantage now while it’s possible.
5. Claude only performs web searches a third of the time
There’s a significant caveat: ChatGPT initiates web searches for nearly 95% of prompts, but Claude does so only a third of the time, likely due to cost ($5 per thousand searches via Brave’s API).
You can optimize Claude effectively only when it conducts a search.
The encouraging part is its predictable search habits. Prompts framed around recent events (“best X in 2026”) initiate searches about 81% of the time.
Ranking-related prompts lead to 67% search initiation, location-specific prompts 55%, and comparisons 51%.
Prompts concerning definitions and procedures rarely trigger searches, making them poor targets for Claude optimization.
The lesson: Before investing to enhance Claude visibility for a prompt category, determine if Claude actually conducts searches for it.
Focus on recent events, rankings, locations, and comparisons for effective Claude optimization using Brave rankings as a guide.
Other areas rely on internal memory beyond our reach.
6. Query fan-out: A raffle on one platform, near-deterministic on another
Two speakers offered perspectives on query fan-out, presenting a contrast worth exploring.
Query fan-out entails background synthetic queries to collect content prior to providing an AI-generated response.
Mike King of iPullRank viewed it as a raffle: The task is to gain more tickets through a wider content range across owned, earned, and shared channels, and the right content formats make all the difference.
Even if you rank for a fanned-out query, the wrong format renders you ineligible.
According to his research, content-to-query cosine similarity and information gain strongly correlate with success in AI search.
Conversely, Josh Blyskal from Profound notes that Claude’s fan-outs are highly predictable; the same prompt results in consistent fan-out strings 65% of the time. Interestingly, 94% of Claude’s fan-outs are current-year stamped, unlike ChatGPT’s 17%
Where ChatGPT’s fan-outs constantly evolve, Claude’s remain relatively stable. Thus, both perspectives may hold true for different engines.
With stable fan-outs like in Claude, content creation can directly focus on them. The year-stamping trend suggests using the current year in titles.
For volatile fan-outs as in ChatGPT, King’s approach applies: maximize exposure through format variety.
One mechanism demands two strategies, tailored by engine, potentially requiring prioritization between them.
7. The marketing engineer is here, and agents are the new workforce
The role of a “marketing engineer” might sound like a buzzword, but the hiring trends prove otherwise.
Google’s recently hired its first marketing engineer, Figma has an opening at a $295,000 salary, and both RBC and Autodesk have placed hires.
It’s a rapidly growing search term, and Google’s AI marketing lead dubbed it “the hire for 2026.”
What makes someone ideal for this role? Is the priority given to an engineer learning marketing or vice-versa?
The emerging profile emphasizes marketing experiences such as someone with channel expertise who builds and runs AI systems, reports to the marketing head, and supports the team by removing obstacles. They are marketers advancing the state-of-the-art.
The underlying concept is that marketing functions decompose into pipelines: data extraction, transformations, and loading into useful formats. Agents can now automate these pipelines.
Monitoring competitor pricing and auto-generating sales content.
Scheduling and assessing AEO presence and landing page efficiency.
Analyzing sales call objections and drafting relevant content solutions.
What previously were backlogged tasks now become brief agent-building exercises. Creativity replaces headcount as the limiting factor.
If marketing engineering isn’t a role in your team yet, it’s likely only a matter of time before it is.
The job now: Figuring out how this all works
There remains no definitive roadmap for AI search. When a guidebook emerges, the key step will be prioritizing one LLM contingent upon who you wish to reach.
In many instances, that “who” will now be agents, simultaneously assisting us in our endeavors and highlighting the rising need for professionals adept at engineering such systems.
Recently, I delved into an intriguing study exploring how enabling search impacts ChatGPT’s product recommendations. Remarkably, these changes affect a vast 80.2% of responses, as observed from an extensive analysis of 20,000 interactions conducted by Jeff Oxford, the founder and CEO of Visibility Labs.
In Oxford’s experiment, he executed 1,000 product-recommendation prompts, running each ten times with search enabled and ten times with it disabled.
Surprisingly, a mere 19.8% of products recommended without search were repeated in the results with search activated.
Search reshapes top suggestions. Even the products that ChatGPT frequently recommended without search seldom appeared once search was turned on. Among those consistently recommended in search-disabled responses, only 15.8% showed up when search was activated.
Oxford anticipated that highly recommended products would still dominate with search, but they turned out to have the least overlap.
Source mentions and visibility. This study also scrutinized whether products cited in ChatGPT’s sources appeared more frequently in recommendations, showing a modest correlation of 0.4 Pearson between source mentions and recommendation frequency.
Products mentioned more often in cited sources had higher Visibility Scores, based on the percentage of instances a product appeared for a given prompt.
The analysis didn’t prove that source mentions directly caused these recommendations.
Search refines the list. With search enabled, ChatGPT’s responses averaged 5.2 products compared to 6.2 without search.
On average, across ten runs for each prompt, there were 19 unique products returned with search enabled, versus 21.8 with it disabled.
Why it matters to us. These findings are crucial because they show how search significantly changes ChatGPT’s product recommendations, even for staple products. Also, products cited in sources may achieve greater visibility when search is enabled, though this study doesn’t conclusively show that source visibility is more influential than web visibility as a whole.
About the study. The analysis covered 1,000 product-recommendation prompts, with each run ten times with search enabled and ten times without. Product names were standardized for consistency. As an observational study, it didn’t establish a direct cause between source mentions and recommendation frequency.
The detailed report. For more insights, see the full study here.
When it comes to enhancing AI’s view of our content, understanding which topics influence those third-party authority signals is vital. Through personal insights, I’ve learned that AI doesn’t always rely on the same sources for every query. Instead, identifying and engaging with those voices that shape answers in your niche can make a huge difference.
I’ve often been advised to build authority outside my site using digital PR, mentions, and gaining links from high-authority sources to boost AI visibility. While the instinct is right, I’ve realized that these efforts need to be specific to the topic because AI trusts different sources depending on the subject matter.
This week, I’ve been exploring why AI leans on various source sets based on different topics, wasting resources on scattered authority efforts, and how to pinpoint exactly where AI derives its citations for your focus area. By doing so, you can carve out your place among those trusted sources.
AI’s approach is fascinating. It relies on a distinct set of sources for each topic. Through a sample analysis, it became evident that AI citations follow a topical pattern, affecting where I focus my authority-building efforts.
In invoicing-related queries, competitor domains account for a significant portion, whereas for starting-a-business inquiries, those numbers drastically decrease. This shift highlights the importance of a topic-focused backlink strategy, where I aim for links with authority in my target topics.
Interestingly, video and social platforms behave differently, affecting visibility. Entities like YouTube are exceptions, especially across larger language models, creating unique pathways to building authority.
My goal is to strategize where and how I build authority, ensuring it’s topic-driven to resonate accurately within the AI’s trusted data pool. Copying PR strategies from adjacent topics won’t yield effective results; instead, I tailor my approach to align with AI’s preferences.
AI’s genuine trust extends from entities it already acknowledges. This means our brand’s perception by AI doesn’t start fresh every time. Instead, it’s shaped by existing trust in associated authoritative sources and documents.
Therefore, my work extends beyond my blog. While it’s crucial, it’s just one part of the broader picture. Publications and experts that mention me streamline my standing with AI models.
Authority developments don’t spread evenly; they progress in leaps. A quality mention can dramatically elevate your citations, especially within high-authority domains. From my experience, investing in in-depth relationships with top-tier sources pays off significantly more than spreading efforts thinly across lesser-authority sites.
This journey to building authority is strategic and measurable. It’s about making informed choices about who we engage with and understanding that authority-building is not just about quantity but also about the lasting relationships we foster in our professional spaces.
How I’m Building Authority in AI’s Trusted Sources
I realized that not all third-party signals weigh the same. While research picked up across reputed blogs can enhance citation frequencies, executive podcasts may not. Here are some steps I’ve taken:
Firstly, I’ve identified a few willing subject matter experts (SMEs) within our team who have a strong understanding of our brand and are eager to publish. Empowering them to create sharp, relevant content fosters credibility.
Through mapping exercises, I’ve pinpointed entities already respected by AI and targeted collaborations with the people behind them. This targeted approach ensures my SME’s work is recognized by the AI and appreciated by audiences.
3. Enhancing Authority Tier Concentration
By ranking potential partners by their authority tier, we refine our investment strategy, concentrating efforts where they’ll generate the most return. This tactic has consistently improved our citation metrics.
Finally, embracing the power of LinkedIn, I leverage influencer partnerships in my domain. This platform acts as a fast lane, allowing us to penetrate AI responses with reputable voices rapidly.
When I first heard about the Profound Index, it intrigued me as the ultimate leaderboard for AI search. Its reputation precedes it, setting a benchmark for excellence in AI-driven search solutions.
The image above perfectly encapsulates what the Profound Index represents—a fusion of innovation and performance in AI search technology. This impressive leaderboard not only showcases top contenders but also encourages competitive enhancement within the AI community.
For anyone deeply invested or casually interested in AI advancements, understanding the Profound Index provides insights into where AI search is headed. It’s a journey worth exploring for its potential to revolutionize how we interact with and leverage AI search capabilities.
I’ve got some exciting news to share! I’m thrilled to introduce the revamped Profound Index, your go-to leaderboard for AI Search. This update marks a new era in search, providing both clarity and authority in the rapidly evolving world of AI-driven solutions.
In this rebuild, we focused on enhancing the user experience and performance metrics. Whether you’re an AI enthusiast or a professional seeking the latest insights, the improved Profound Index is designed with you in mind. Its comprehensive data sets and intuitive interface make it an indispensable tool for anyone looking to stay ahead in the realm of AI Search.
When I think about how artificial intelligence is revolutionizing advertising, a common belief is that AI is killing advertising. But, in reality, AI is not the end of advertising; it’s merely transforming it into new dimensions. With AI seamlessly integrating into search, assistants, productivity tools, and beyond, it’s only natural for advertising to follow suit.
I’ve noticed that while the density of ads may shift in AI-led experiences, the opportunities for advertising are actually broadening. There are new surfaces emerging continuously, and they all offer exciting chances for advancers and advertisers alike.
To me, the divide between paid and organic isn’t as clear-cut anymore. The same AI systems powering search experiences are also driving ad campaigns and influencing brand visibility across Google’s expansive ecosystem.
This calls for a change in how we brands perceive visibility. Paid and organic aren’t just isolated competitors vying for clicks; instead, they’ve become alternative strategies influencing the same AI systems. As a result, the signals that shape organic visibility may also impact paid performance.
The traditional search engine results page (SERP) we once knew, consisting of 10 blue links, a handful of ad slots, and a side panel, no longer holds the same dominance. Back then, dedicated teams managed paid and organic strategies separately, each with its own set of tools and quarterly goals.
Things changed for me when Dynamic Search Ads (DSA) appeared, using my website’s content to cleverly create ad titles and determine bids, merging the lines between our organic strategies and paid efforts.
Stepping into the modern age, Performance Max (PMax) campaigns took the very logic of DSAs and applied it across every Google surface—importantly altering how ads are placed from Search and YouTube to Maps and more.
Of course, it isn’t without its nuances. If Google’s Gemini doesn’t have a thorough understanding of our brand, the system has to fill the gaps with assumptions, which may not align with our intended brand narrative. It’s crucial to train these AI systems deliberately, or we risk losing control.
Strategically, I’ve come to realize that paid campaigns help me discover which target audience-intent-profit combinations convert best. I can then build my organic content around these successful elements, creating a feedback loop where each strategy amplifies the other.
Recovering from a manual action is no quick fix; it can take months of rigorous cleanup and multiple reviews. I’ve learned that regular compliance audits are key to avoiding a crisis altogether.
Google penalties—or manual spam actions—are those unpredictable disruptions that can shake up a thriving online business overnight.
For businesses like mine that rely heavily on organic traffic, the impact is quite severe. It goes beyond just losing rankings; revenue takes a hit, customer acquisition costs spike, expansion plans are halted, and the effects linger long after the policy issues have been addressed.
With Google’s consistent 90% market share, it remains my main source of traffic, much like it is for many publishers, e-commerce platforms, and lead generation companies.
Unfortunately, direct traffic seldom makes up for significant visibility losses, and Bing isn’t enough to fill the gap. This means a manual spam action is not just an SEO risk but a grave operational concern.
Manual Actions Aren’t Algorithm Updates
It’s essential for me to clarify that manual spam actions and algorithmic updates are two different beasts. Manual penalties result from specific violations identified against Google Search Essentials and demand entirely different responses.
Manual actions involve considerable internal review at Google. When violations are suspected and verified, these actions are taken, because proven policy breaches aren’t taken lightly by Google.
The real issue lies in recognizing accumulated policy violations over time, something I’ve seen many businesses fail to address adequately.
How Penalties Develop
The journey to a manual penalty often begins in non-obvious ways, with compliance erosion happening gradually.
An e-commerce company might start with aggressive link-building strategies that accumulate unchecked spam links over the years.
A publisher engages in commercial partnerships involving sponsored content, integrating these into their main site structure.
A SaaS business expands into new markets with low-quality location pages.
Lead generation companies scale supplemental SEO content without thorough editorial oversight, simply adhering to industry standards.
Though these tactics might initially boost visibility and revenue, they often fall out of line with Google’s quality standards over time.
Why Historical Violations Still Matter
Manual spam actions are disruptive partly because old policy violations can persist without being flagged for years. Google doesn’t forget historical footprints in its search system, meaning unresolved past SEO practices can become today’s liabilities.
Practices like paid placements, commercial guest posting, or directory spam from years ago can remain risks until they’re addressed, creating vulnerabilities that must not be ignored.
Reputation Abuse and Publisher Liability
When a trustworthy brand allows unsupervised content from third parties, the site’s credibility might suffer. Once a manual spam action hits, the entire site can lose visibility—even the genuinely valuable sections suffer.
Recovery from such penalties is not simple or cheap. It often demands structural changes and more stringent editorial and technical controls, as I can attest from my own experiences.
The Risks of Scaled Content
Google is now more vigilant about large-scale publishing systems that lack originality and value. I’m aware of how easily businesses, unintentionally, slide into creating repetitive, low-value content.
Most businesses don’t cross these lines deliberately. However, without ongoing reviews and updates, significant issues can fester under the radar.
Compliance Requires Ongoing Oversight
For me, regular compliance reviews are non-negotiable. It takes external expertise to assess true compliance comprehensively. Even powerful internal SEO teams can miss potential exposure points if left unchecked.
I’ve found that organizations integrating compliance into governance see considerable advantages. Regular audits and assessments can preempt violations and protect critical search traffic, especially during pivotal business moments.
In essence, prevention through regular audits is a more efficient and less painful approach than dealing with recovery after a penalty.
Today, as I explore updates from Microsoft, I’m excited to share that Bing Webmaster Tools is rolling out a preview of its new AI performance report enhancements. These include insights like Intents, Topics, Citation Share, and Compare, and they’re being introduced globally. After witnessing Microsoft’s demo in April, it’s thrilling to know these features are finally accessible to us.
Reflecting on their past roll-outs, Bing officially launched its AI performance report earlier in February, a bold move ahead of Google’s similar feature which wasn’t available in Search Console until June. Google’s delayed release felt quite rushed to many of us.
New Insights: Krishna Madhavan from Microsoft describes these updates as built to give publishers a clearer understanding of why their content surfaces, the broader subject areas they’re gaining visibility in, and how their presence compares with other sources over time.
Intent: The Intents feature now classifies grounding queries into categories such as Informational, Commercial, Navigational, and more. This provides deeper insight into the intent behind user queries, moving beyond just triggering citations to understanding broader query contexts.
An example given was an e-commerce publisher finding visibility in comparison-focused experiences or an educational publisher learning their content is popular in research-oriented interactions. These insights can guide us in refining content structure and depth.
Topics: Topics group related queries into thematic clusters, offering us a more organized way to understand visibility, similar to modern AI’s reasoning across themes rather than isolated keywords.
For instance, queries like “solar panels” and “solar energy efficiency” can all be part of a larger topic cluster such as Solar Energy. This thematic organization helps us align our content with how AI systems engage with our content.
During this preview phase, some labels might remain broad, especially for niche domains, but meaningful patterns are already emerging.
Citations: Citation Share now displays the percentage of citation visibility your site enjoys compared to others. It’s a directional metric to help us understand our evolving visibility over time, without ranking or quality scores.
Compare: We can now compare citation changes over time. This feature overlays previous data onto current reports, helping us observe citation activity, which can be influenced by various factors like AI model updates, user demand shifts, and more.
Why This Matters: Although we’re still awaiting click and click-through rate data, these growing AI performance insights are invaluable. I’m hopeful that such detailed data will become available to us from Microsoft or even Google one day.
I’ve witnessed AI tools become indispensable in automating complex processes that traditionally demanded a lot of manual effort. However, I’ve also seen them used without any real benefit just because they are available.
That’s why I prefer focusing on AI applications that save time and address genuine challenges.
Recently, I was tasked with aligning the SEO architecture for over a dozen websites across three separate businesses, eight regional domains, and numerous languages, including three English dialects, Italian, Japanese, Spanish, Thai, French, and Korean.
Mapping thousands of URLs to create seamless hreflang XML sitemaps traditionally required specialized software or extensive spreadsheet work. Instead, I used Google Gemini to develop a custom Python script to handle the heavy lifting.
Here’s how an initial prompt evolved into a fully customized automation tool and what it taught me about utilizing AI for technical SEO.
Where AI Delivers the Most Value
I leverage AI primarily for practical, time-saving tasks, including:
Generating regex patterns when I need quick solutions without researching syntax from scratch.
Creating complex spreadsheet formulas for reporting workflows that depend on manual data exports.
Speeding up research and planning for projects requiring competitive analysis across business lines.
Building custom automation tools for recurring SEO and data-processing tasks.
The hreflang project I discuss here fits perfectly into the last category.
Mapping hreflang at Scale
The challenge was straightforward: accurately map thousands of URLs across multiple multilingual websites into cohesive hreflang XML sitemaps.
I chose not to tackle this manually. Instead, Google Gemini helped me build a custom Python solution.
Here’s a walkthrough of how the process unfolded.
Phase 1: Asking for an Approach, Not Just a Script
One common pitfall of using generative AI for coding is asking it to sprint before understanding the course. Typing, “Write a Python script to create an hreflang sitemap,” will yield generic code prone to breaking with real-world data.
Instead, I started by asking for an approach. I detailed the scenario: multiple regional domains, organic growth over several years leading to mismatched URL slugs, translated subfolders, and appended revision years.
Gemini suggested a multi-step, data-driven approach:
Crawl the websites to collect live URLs and their metadata.
Use Python in Google Colab to process the raw data.
Run an exact match cluster to group identical slugs.
Use an advanced semantic AI model (like SentenceTransformers) to fuzzy match translated pages based on their titles and normalized URLs.
Phase 2: Crawling and Data Collection
Following the recommended strategy, I used a crawler to spider all regional websites to generate a unified CSV file with live URLs, status codes, title tags, and H1s. Screaming Frog proved ideal for this task.
The quality of AI output relates directly to the quality of your crawl data, so make sure it’s robust.
An AI script can miss an obvious “exact match” if a target URL is a 404 or a 301 redirect. Before feeding data into the script, filter your CSV to include only indexable content.
Google Colab offers a free, cloud-based Jupyter notebook environment for coding, bypassing local installations or environment variable issues. I used Google Drive to access it. The free version sufficed for this project.
After uploading the CSV to Colab, Gemini provided an initial Python script that utilized a domain-mapping routine to assign language codes, clean the URLs, and generate an XML tree. The initial results required refinement.
Phase 4: The Iteration (Where the Real Work Happens)
If you expect AI to produce a flawless script on the first try, you’ll be disappointed. Like an intern, AI requires oversight. The true value lies in iteration.
After running the initial script, several unmatched URLs left orphaned pages rather than grouping them with international counterparts. Here’s how I iteratively guided AI through the complexities of human-managed websites.
The Directory Flattening Problem
The U.S. site had recently reorganized its blog into topical folders, unlike the Mexican and Italian sites. I presented these mismatches to Gemini, leading to a script adjustment that flattened directories, allowing slugs to align.
The Aggressive Semantic Trap
Concept traps we implemented were initially strict. A UK article about manufacturing wouldn’t match its Italian counterpart due to a slightly different title. By loosening these traps for general industries and enforcing them for critical terms, the AI became adept at delivering better matches.
The Translated Slug Epiphany
The pivotal insight arrived when examining Mexican blog orphans. A Spanish URL /detras-de-escenas-historias... matched the English /behind-the-scenes-stories..., which I pointed out to Gemini. As a result, Gemini updated the script to create a “Combined Semantic Signature,” dynamically translating slugs and efficiently bridging language gaps.
This project reinforced a simple truth: AI excels as a collaborator rather than a shortcut.
Be the strategist, let AI be the coder: Rather than demanding a finished product, discuss architecture and logic first, treating AI as a junior developer needing guidance.
Provide concrete examples: Don’t simply state, “It’s broken.” Give specific failed URL examples or mismatches to help AI refine its logic.
Embrace the iterative loop: Run the code, identify issues, and iterate. Each iteration enhances the tool’s intelligence.
Leverage Google Colab: You don’t need to be a Python guru to apply Python in SEO. Colab bridges the gap, providing access to complex data science libraries in your browser.
In the end, I had a fully customized Python script capable of processing a massive CSV to generate a cross-referenced hreflang XML sitemap in minutes.
Though AI isn’t replacing technical SEOs, those who collaborate with AI to build scalable tools will have a significant edge.
