I’ve always been fascinated by how Google keeps improving its search capabilities. Recently, Gary Illyes from Google shared more about Googlebot’s operations, diving into its crawling ecosystem, fetching processes, and how it handles data.
Googlebot Reimagined. It’s intriguing to learn that Google uses multiple crawlers for diverse objectives. Referring to Googlebot as a singular entity might not capture this complexity anymore. You can find more details on the various crawlers and user agents here.
Understanding Limits. During a recent discussion, Google elaborated on its crawling limits. Gary Illyes provided these insights:
Googlebot fetches up to 2MB for any individual URL, except for PDFs.
This means it crawls only up to 2MB of a resource, encompassing the HTTP header.
For PDF files, the limit is notably higher at 64MB.
Image and video crawlers have varied threshold values, contingent on the product they serve.
By default, other crawlers have a 15MB limit, regardless of content type.
What exactly occurs when Google initiates crawling?
Partial Fetching: For HTML files exceeding 2MB, Googlebot will not dismiss the page. Instead, it halts the fetch exactly at the 2MB mark, including HTTP request headers.
Processing the Cutoff: The downloaded section is then forwarded to Google’s indexing systems and the Web Rendering Service (WRS) as if it were the entire file.
The Unseen Bytes: Any data beyond the 2MB cutoff won’t be fetched, rendered, or indexed.
Resource Handling: All referenced resources in the HTML, except media, fonts, and certain files, are fetched by WRS independently, with their own byte count not affecting the parent page’s size.
Rendering Bytes with Google. Once the crawler accesses these bytes, WRS takes over. It processes JavaScript and executes code like a modern browser to grasp the final visual and textual state of the page. This process doesn’t request images or videos but does respect the 2MB threshold for each resource.
Best Practices You might want to embrace these recommended practices:
Streamline Your HTML: Shift large CSS and JavaScript to external files. While the main HTML document is capped at 2MB, external scripts and stylesheets can be fetched separately, under their own constraints.
Prioritize Content: Position crucial elements like meta tags, <title>, <link>, canonicals, and vital structured data high in the HTML to ensure they’re not overlooked.
Monitor Server Logs: Keep track of server response times. If your server struggles to deliver data efficiently, our fetchers may slow down to avoid overloading, reducing crawl frequency.
Don’t Miss the Podcast! Google also released a podcast on this topic. Check it out:
Diving into the world of technical SEO for generative search has had me rethinking how AI agents interact with my site. It’s not just about indexing anymore; it’s about how AI systems generate answers. My focus is now on ensuring AI agents can access and interpret my content smoothly, enhancing the chances that I’ll be cited in AI-generated responses.
When I consider generative engine optimization (GEO), I’ve realized that while the underlying tools and frameworks aren’t new, the way I implement them makes the difference in my content being surfaced or missed.
It means paying close attention to how AI agents access my site, structuring my content for easy extraction, and ensuring it can be reliably interpreted and reused in AI-generated responses. This is about precision and strategic structuring.
Agentic Access Control: Managing the Bot Frontier
Using robots.txt strategically has become vital. It’s essential for me to specify which crawlers can access what parts of my site. For instance, I might decide that a training model like GPTBot should access my /public/ folder but keep my /private/ folder off-limits, implementing it as follows:
The choice between model training and real-time search is crucial. Often, I find myself balancing whether to disallow GPTBot or allow OAI-SearchBot. Considering Perplexity and Claude standards within my robots.txt is another layer I need to manage:
Claude
ClaudeBot (Training)
Claude-User (Retrieval/Search)
Claude-SearchBot
Perplexity
PerplexityBot (Crawler)
Perplexity-User (Searcher)
I’ve also had to integrate the new protocol, llms.txt. Although not universally adopted, it’s a structure I find useful for guiding AI agents in understanding my content better. If you’re interested in following Perplexity’s llms.txt, you can explore it here:
llms.txt: A concise map of links.
llms-full.txt: An aggregate of text content that allows agents to bypass crawling my entire site.
Even if Google and others aren’t reading llms.txt right now, I believe it’s worth preparing for future needs. John Mueller has shared insights on this which you can read here.
Extractability: Making Content ‘Fragment-Ready’
In the realm of GEO, I’ve been focusing on creating content fragments because AI systems value precise and concise information. This means avoiding bloated content that can hinder AI retrieval due to issues like:
Challenges with JavaScript execution.
Overreliance on keyword optimization instead of entity optimization.
Poor content structures lacking clear answers.
To make my core content visible and accessible to various AI entities, semantic HTML components like <article>, <section>, and <aside> have become essential tools. This separation helps the essential facts stand out, feeding search engines and AI bots effectively.
Technical SEO is evolving, and as I adapt, I’m focusing not just on visibility, but on becoming a source of truth for the world’s AI models. By using structured data efficiently, implementing robust access control via robots.txt, and refining my content’s extractability, I’m setting the stage for success now and into the future.
Ensuring my strategies are working requires thorough auditing. I focus on areas like citation share, log file analysis, and zero-click referrals to measure how effectively my content is influencing the AI-driven world. This helps validate my efforts and enhance KPIs.
Scaling GEO into 2027
Looking ahead to 2027, I’m prioritizing automation to minimize manual optimization work. The goal is to leverage every SEO tool available, ensuring my site is a robust source of truth amid AI advancements. Starting with basics like robots.txt and moving towards more sophisticated structures, my ongoing goal is to scale efficiently and effectively.
I’ve noticed a significant shift in the SEO industry toward senior, strategy-focused roles. As AI increasingly handles execution tasks, the demand for seasoned strategists has grown, along with an increase in salaries and responsibilities that span multiple channels.
The change in hiring trends is evident when looking at a recent Semrush analysis of 3,900 job listings. It appears companies are now prioritizing leadership skills, innovative experimentation, and cross-channel visibility over purely technical execution.
Why it matters to me. The landscape for SEO careers and skillsets is evolving. Entry-level positions are mostly focused on execution, while leadership roles require a firm grasp of strategy across various domains such as search, AI assistants, and paid channels, ensuring they drive significant revenue.
What’s changing now. Senior roles account for 59% of job listings, clearly dominating the landscape. In contrast, mid-level positions like specialists and managers are less prevalent, with only 15% and 10%, respectively.
Companies are redirecting their budgets towards strategic roles as AI tools begin to absorb more of the technical workload.
The shift in skills. The skills in demand now extend beyond traditional SEO to include coordination, experimentation, and decision-making capabilities:
Project management is mentioned in over 30% of the listings, highlighting its importance.
Communication is highlighted in 39.4% of non-senior roles, indicating its fundamental role in the industry.
Experimentation is noted in 23.9% of senior roles, compared to just 14% of other roles.
Technical SEO appears in approximately 6% of postings, showing its niche but crucial role.
Tools and channels. The modern SEO toolkit now includes analytics, paid media, and comprehensive data tools.
Google Analytics is cited in up to 47.7% of job listings, underlining its importance.
Google Ads features in 29% of the listings, showcasing its growing relevance.
Demand for SQL skills is rising, especially at the senior level.
AI tools, such as ChatGPT, are increasingly mentioned, reflecting their future role in SEO.
AI expectations. AI literacy is shifting from being a nice-to-have to an essential skill:
31% of senior roles now reference AI capabilities.
Nearly 10% of listings highlight familiarity with LLMs.
Concepts such as AI search and AEO are increasingly common in job descriptions.
Pay and positioning. SEO is being increasingly recognized as a vital business function:
The median salary for senior roles has reached $130,000, markedly higher than the $71,630 for other roles, with some positions offering even more.
Preferred degrees are leaning towards business and marketing, reflecting the strategic emphasis.
Remote work prevalence. Remote options are available in over 40% of job listings, indicating a shift towards flexible work environments across all levels.
About the data. This analysis by Semrush covers 3,900 SEO job listings in the U.S., gathered from Indeed as of November 25. The roles were deduplicated and segmented by seniority before a semantic keyword extraction analysis was applied.
In 1998, I found myself meticulously submitting websites to search engines. I remember the drill well: AltaVista, Yahoo Directory, Excite, Infoseek, Lycos, and others. Each had its own form and wait time, leaving us to wonder if our URLs would make the cut.
Back then, we submitted a whopping 18,000 pages, manually. While this was happening, Google was just emerging. Yet, they already had a vision that would render manual submissions almost obsolete.
Google’s PageRank meant that if a site had incoming links, it didn’t necessarily need to submit. While other search engines waited, Google proactively discovered content, streamlining what was once a tedious process.
For two decades, the rule was simple: you published, you waited, and the bots would come. But now, the landscape is shifting. Not because Google has lost its edge, but due to an expanded game where merely waiting won’t capture all available revenue streams.
The pull model, which depends on search bots, is no longer the only method of content discovery. We now have five modes of entry into the AI engine pipeline, and the single entry mode of the past has evolved dramatically.
I’ve identified these modes to show how they each confer unique advantages at the crucial stages of indexing and annotation, which determine a content’s competitive edge.
First up, the traditional pull model remains, where bots fetch and decide everything. It offers no structural leverage, leaving content entirely dependent on the bot’s schedule.
Next, push discovery is a proactive approach, notifying systems of new or updated content. Tools like IndexNow by Bing expedite this process significantly, allowing content to be recommended much sooner.
Push data skips the bot entirely, using structured data to directly feed AI systems. Here, seamless indexing from a machine-readable format offers a major competitive edge.
Push via MCP allows AI agents to access real-time data directly, transforming how content enters the competitive arena. Brands without MCP-ready data risk losing out to those with real-time access capabilities.
Finally, ambient entry is about AI recommending content without explicit user queries, often seen in tools many of us use daily.
All modes converge at the annotation phase, a critical step for successful content visibility in AI systems. As we shift focus on entity management and centralized data, brands can optimize for all entry modes, ensuring readiness for any future developments.
Welcome to the ultimate guide on Generative Engine Optimization (GEO)! As we move into 2026, knowing how to optimize for AI-driven platforms like ChatGPT, Gemini, Perplexity, and Claude is crucial. This guide will help you ensure that your brand is easily discovered in AI-generated responses.
Imagine having the skills to make your brand the first choice for AI-powered searches. With our comprehensive insights, you’ll learn how to elevate your visibility across key AI platforms and gain a competitive edge.
Whether you’re a seasoned marketer or new to AI optimization, this guide offers strategies that align with both current trends and future predictions. By mastering Generative Engine Optimization, you’re setting the foundation for sustainable success in a rapidly evolving digital landscape.
I’ve got some exciting news to share—Reddit has just opened up its Pro publishing tools to all publishers! No more waiting lists. Now, anyone can dive into the public beta and ramp up their content distribution and engagement strategies, all for free.
Why this matters to us. Reddit Pro offers me a centralized hub to monitor where my content spreads, simplifying my posting process, and helping me pinpoint the right communities to engage with. It’s transforming Reddit from being a place of manual posting to a well-organized distribution channel.
Here’s the scoop. I can now easily sign up for Reddit Pro, verify my domain (usually within three business days), and jump into the Links tab. With Reddit Pro, I can:
Keep track of where my content is shared all over Reddit.
Quickly auto-import articles through RSS, speeding up my posting.
Receive AI-powered tips on the most relevant communities to connect with.
Reddit has also rolled out some features based on early adopter feedback:
Community snapshots that display rules, stats, and top discussions.
Community notes that let me track strategy and context over time.
By the numbers. Back in 2025, Reddit revealed there were over 55 billion views of publisher-related discussions. Since some publishers started testing in September, they saw:
A 46% uptick in median post views.
An almost doubled amount of profile views.
A 48% climb in median comments.
What else to look forward to. Reddit is also expanding profile flairs to every Pro user. This means I can organize posts on my profile, making it easier for users to browse my coverage and get involved with stories.
As someone who closely follows advancements in technology, I was thrilled to learn about Google’s latest breakthrough with the TurboQuant algorithm. It’s designed to enhance the speed of vector searches, fundamentally changing the way we interact with AI-powered data searches.
If you’re like me and value precision in data retrieval, this algorithm is exciting news. A tiny error-correction signal maintains compressed vectors’ accuracy, enabling AI systems to retrieve data more broadly and precisely than ever before.
Google’s TurboQuant is a compression algorithm that can shrink and organize large AI datasets with nearly zero indexing time. This technology might just obliterate one of the major speed bottlenecks in modern search engines.
What TurboQuant Is. For me, TurboQuant represents a monumental way of handling the data behind AI and search by keeping it compact without losing precision. It significantly reduces memory usage and cuts down the time to build searchable AI indexes almost to zero, according to Google’s research paper.
How It Works. Modern search systems, which convert content into vectors, can be resource-heavy. These numeric representations cluster based on similarity, allowing searches to match the closest ideas. But let’s face it, these vectors are massive and expensive to store. That’s where TurboQuant steps in, using efficiently compressed data that mirrors the original extremely well through:
Smart Compression. It rotates data mathematically, organizing it like neatly packed boxes, an image that resonates with how I like to visualize innovative data solutions.
Error Correction. By introducing a 1-bit signal, it corrects minor compression mistakes, ensuring the data remains accurate, which is quite a comforting thought for anyone concerned about data integrity.
What This Means. For those of us deeply engaged with AI, TurboQuant signifies a shift. Vector search systems, the backbone of semantic search and AI-driven answers, have traditionally been slow and costly. Google claims TurboQuant makes these operations quicker and more cost-effective, enabling faster similarity searching, lower memory consumption, and real-time processing of colossal datasets.
Why It Matters to Us. Imagine Google being able to analyze far greater volumes of documents per query, not just a limited subset. Should Google implement this into its Search, AI Overviews could access a wider, more accurate range of sources, making instant summaries from large data sets far more accessible.
Have you ever wondered why most GPTs in businesses fail to be truly effective? It’s often because they are either too broad or haven’t been properly tested. Allow me to guide you through building focused, high-ROI GPTs that your team will not only adopt but use consistently every week.
The OpenAI GPT Store made waves in January 2024 with its launch, hosting over three million custom GPTs. But, if you ask teams how many they actively use, the answer tends to be disappointingly low, often zero or just one.
I’ve personally built and audited over a dozen custom GPTs for marketing, SEO, and sales teams. The pattern is consistent: only a select few are used daily, while the rest simply collect dust. Let me share with you a practical approach to crafting GPTs that your team will genuinely engage with—from identifying suitable use cases to structuring, testing, and launching them effectively.
If you’re eager to dive in, start with these foundational steps: Choose a task your team performs at least three times a week, typically taking over 15 minutes. Articulate this in a simple sentence: ‘This GPT helps [role] do [task] by [method].’
For a deeper understanding, I recommend checking out Marketing Research & Competitive Analysis or MARKETING, both highly ranked in the GPT Store’s Research & Analysis category. These projects showcase the build patterns I’ll cover here.
Now, let’s discuss what a business GPT truly entails. Unlike a generic AI assistant, a business GPT is a custom version of ChatGPT designed to handle one specific, recurring task for a particular role. Think of it like hiring a highly specialized worker for a job, rather than a generalist who does a little bit of everything.
I recently explored what many are calling the ‘Global Spanish’ issue in AI search visibility, and it’s been a revelation for understanding how AI can sometimes blur crucial distinctions in Spanish-speaking markets.
Picture this: AI models often clump Spanish-speaking regions into one, mixing up local jargon, regulations, and context, resulting in answers that don’t truly fit any specific market.
This challenge—commonly known as the ‘Global Spanish’ problem—manifests when AI search merges regional dialects and rules into a one-size-fits-none guidance.
Consider asking AI in Spanish how to declare your taxes (cómo puedo declarar impuestos). It will deliver a grammatically accurate reply, equipped with references like ‘RFC, NIF, SSN, según país’—mixing up Mexican, Spanish, and American tax identification.
While AI is gradually improving, moving from confidently incorrect Mexican tax advice in Madrid to a more hedged but jumbled response doesn’t equal localization. It’s more like broad-stroke thoroughness without precision.
The core issue is AI’s struggle to pinpoint its targeted Spanish-speaking market, defaulting to overly generalized responses akin to a waiter asking a roomful what they’ll have and simply writing down ‘Food.’
If I find that AI answers a Mexican with Spain’s tax logic, this isn’t just a translation hiccup—it’s a fundamental problem with geographical and jurisdictional inference, essential in AI-facilitated search.
Traditional search already faced these complexities, and giants like Google spent years refining systems to accommodate regional intent and language variations—challenges that persist today.
Generative AI, however, eliminates the wiggle room. Instead of multiple links allowing user choice, it delivers one synthesized answer, hitting home or missing the mark entirely.
For many, ‘Spanish’ is a simple language toggle, but this view doesn’t hold for Hispanic markets. The distinctions between Spain and Latin America go beyond slang; they influence conversion rates, brand trust, and legal applicability.
Cultural and regulatory differences exist, such as:
Regulators like Hacienda vs. SAT.
Legal terms such as NIF vs. RFC.
Currency differences, such as EUR vs. MXN.
Decimal formatting like period vs. comma.
Tone variation for social distance (tú/vosotros vs. usted/ustedes).
Commercial expectations like payment options and shipping norms.
Search intent, where identical queries target different products depending on the country.
All these affect international SEO, and in generative search, they become critical. The AI doesn’t present multiple links for user discretion; it condenses everything into a singular, presumptive authoritative answer, leading to what I recognize as ‘Global Spanish.’
Studies term this bias as ‘Digital Linguistic Bias’ (Sesgo Lingüístico Digital), revealing how imbalanced Spanish variety representation in corpora ignores dialectal variations and cultural contexts due to structural bias.
Spain, despite being a minority among global Spanish speakers, is overly represented in digital resources guiding language models’ default Spanish. Latin America, conversely, is under-represented in AI investment and data infrastructure, with just 1.12% of global AI funding while contributing 6.6% of global GDP.
This naturally skews AI-produced Spanish towards sounding geographically particular, despite users not specifying a region. Because LLMs train on the most available web data, which often disproportionately represents certain locales, this bias emerges.
A Mexican SaaS webpage, excellently drafted, competes against decades-old Peninsular Spanish content for AI’s attention and often loses, with ‘neutral Spanish’ considered efficient but ultimately impeding the scale.
These shortcomings manifest as three distinct failure modes, each critical to SEO results, trust, and conversion rates.
1. Dialect Defaulting: Often AI defaults to one Spanish variant, misleading users from other regions.
Tested by Will Saborio, terms like ‘straw’ varied across countries—’pajilla,’ ‘popote,’ ‘pitillo,’ and ‘bombilla’—but AI typically defaulted to Mexican Spanish. Even detailed prompts for Colombian content didn’t localize the results consistently, a pattern echoed by studies evaluating multiple LLMs.
Dialects involve vocabulary, product categorization, idioms, formality, and embedded cultural assumptions. A product page coded for Spain can alienate a Mexican user, with AI further reinforcing that outsider signal.
2. Format Contamination: Incorrect formats silently harm conversions, like a presence showing local format as incorrect.
An issue documented in Unicode ICU4X shows Mexican Spanish uses periods as decimals, whereas default data might unintentionally apply European format, switching periods and commas. This leads to misinterpreted values e.g., 1.250 could mean one thousand two hundred fifty or one-point-two-five-zero based on locale defaults, which I have personally experienced with damaging mispricing for localized Black Friday deals.
3. Legal and Regulatory Hallucination: AI errors in legal content can be detrimental to YMYL content, reducing Google’s E-E-A-T signals.
Minority Spanish-speaking countries have distinct legal contexts; reporting incorrect legal framework advice can breach regulations, risking being omitted in AI answers.
These issues highlight a pivotal AI geo-identification misstep: language is treated as a geographical hint. Without explicit signals, AI answers hover between multiple locales like Mexico, Spain, or Colombia, lumping distinct markets into ambiguous responses.
Take for instance Blas Giffuni’s example of ‘proveedores de químicos industriales’—chirping back U.S. suppliers rather than Mexican relevant ones—showing geo-drift as AI mistakes linguistic tasks for informational needs.
This is a pressing issue as Spanish AI-driven search visibility scales up, with Google’s AI Overviews rolling out across Spain, Mexico, and Latin countries, serving summaries often drawing from ‘generic Spanish,’ quite possibly eclipsing local terminology and legal references.
Even with localized content prepared methodically, AI’s skewed training models amplify English over Spanish, perpetuating an idealistic U.S.-centric view as highlighted by Pieter Serraris through log analysis, showing AI preferring English corpus significantly more frequently than foreign counterparts.
Additionally, tokenization taxes raise the cost of conducting AI tasks in Spanish due to longer word structures compared to English, leading to higher APIs bills along with limiting crucial context windows.
Moreover, English domains intrinsically pick up stronger authority signals and wider reach causing retrieval bias, progressively edging out localized Spanish sites which slowly descend into digital obscurity.
This shifts SEO priorities from simply ranking pages to modifying entity perception within AI frameworks, contrasting SEO’s traditional approach. The key takeaway is ensuring explicit context conveying where content belongs linguistically and geographically, becoming critically essential in this new generative search landscape.