I’m excited to share that Microsoft has launched a groundbreaking search service specifically designed for AI-agents, as agents have unique search requirements compared to humans.
I’ve learned that Microsoft’s latest innovation, Web IQ, is here to bridge AI systems with real-time intelligence online. As a suite of AI-native grounding APIs, Web IQ sources fresh data, be it web pages, news, images, or videos, as announced by Microsoft here.
What is Web IQ? Web IQ is all about connecting AI systems to real-world updates, leveraging Bing’s index for superior understanding. I find it fascinating how it uses the same infrastructure as Microsoft Copilot and other leading LLMs, like ChatGPT.
However, I discovered that Web IQ’s APIs are newly developed for efficiency and relevance, crucial for serving Bing, Copilot, and ChatGPT queries rapidly.
For AI-Agents, Not Humans. Web IQ tailors search results specifically for AI-agents. Unlike human-oriented Bing Search, ranking isn’t a priority here, as agents need swift information extraction, as stated by Jordi Ribas, President of Search & AI at Microsoft.
Unlike us, AI-agents don’t just issue a single query; they delve deeper and continuously expand their search. This paradigm shift meant re-architecting search from indexing to orchestration, aligning it with AI needs, as per Microsoft’s insights.
Given the frequency of searches AI-agents perform, Microsoft designed Web IQ to operate efficiently, minimizing token usage to deliver better and faster results. It’s currently 2.5 times faster than its nearest competitor.
Access and Availability. At present, Web IQ supports Microsoft Copilot, OpenAI’s ChatGPT, and other large LLM platforms. As Microsoft scales this technology, I expect wider access to follow.
If you want to express interest in Web IQ, Microsoft encourages you to visit this page.
Why this Matters. As we witness the web transforming to accommodate agentic technologies, keeping an eye on these developments is vital. Websites, including mine, must evolve alongside these AI advancements.
AI-agents aren’t just a trend; they’re part of the web’s next evolution. I’m preparing to embrace this change, and I suggest you do too.
I often find myself grappling with the essential need to make sure AI-generated work aligns with our brand’s identity and campaign history. This is where a structured ‘client brain’ comes in handy, providing context that grounds AI in brand guidelines and technical nuances.
Each SEO agency I know deals with what I like to call a ‘context tax.’ It’s the unspoken burden when strategists and analysts have to remember intricate account details like brand voice, past keyword decisions, CMS limitations, and what angles the founder disliked.
The challenge with integrating AI into complex SEO tasks is giving it enough context to be genuinely useful. One approach I’ve been exploring is the concept of a ‘client brain,’ a system that retains account-specific knowledge, allowing AI to act with the intelligence of someone who’s been involved from the start.
Context is the Problem
Understanding context is crucial for any successful worker or AI. As a senior SEO lead, I onboard new team members by sharing vital details about client preferences, past strategies, and technical constraints. Without this, even AI struggles to deliver effectively.
In SEO, we’re increasingly focused on data integration—bringing together metrics from various sources to have a comprehensive view. However, AI isn’t just about data. It’s about using that data in the context of what we’ve learned from the client so far.
I’ve realized that having a centralized repository of client-specific insights—what I consider ‘institutional memory’—is indispensable. It helps our AI avoid suggesting ideas or strategies we’ve previously dismissed or accepted.
A Client Brain is the Solution
Creating a ‘client brain’ means systematically recording all critical decisions, feedback, and client idiosyncrasies. It’s not a substitute for human intuition but rather a framework that aids in applying that intuition consistently across teams and tasks.
In my experience, effective SEO requires multiple hands—strategists set the path, content leaders draft plans, writers create, and analysts evaluate performance. Without shared context, each transition becomes a potential point of drift, where critical details can be lost.
What a Client Brain Is
A client brain is essentially a detailed, organized knowledge base that AI refers to before launching into any task. I like to think of it as carving out the soul and memory of an account, enabling AI to understand both stable brand tenets and evolving client experiences.
Not all knowledge is created equal. Some parts remain constant, like brand identity and audience details, while others evolve, such as campaign outcomes or technical limitations. It’s vital to separate these layers to ensure clarity and usability.
In practical terms, I categorize this into two layers: the ‘soul,’ which includes static brand knowledge, and the ‘memory,’ which documents dynamic experiences and learning moments from working with clients.
The Technical Anatomy of a Brain
I aim for simplicity when building a client brain. It starts as a collection of plain-text Markdown files, eliminating the need for complex systems or special software. The organization into a ‘soul’ and ‘memory’ folder structure helps keep everything manageable.
Building Core Logic of the Soul
To initiate this process, I recommend creating a directory named ‘brain/soul’ and populating it with core files: company profile, style guide, audience, keyword map, and a ‘never do’ list. Each serves to capture essential client insights succinctly.
Within the ‘brain/memory’ folder, I record decisions we made and their rationales, identify recurring patterns in our work, and maintain a chronological log of notes and client interactions. It’s invaluable for maintaining institutional knowledge over time.
brain/memory/
├── decisions/ — choices made and why
├── patterns/ — things that worked or didn’t, by task type
└── log/ — chronological notes by date
Documenting the logic behind decisions is as critical as the decisions themselves. It ensures AI can align with evolving strategies, adapting as contexts change over time.
It’s crucial to start small, choosing the client where losing context costs the most time. Long-running accounts with a distinct brand voice and a history of ideas are ideal candidates.
Step 2: Block 90 Minutes and Write the Soul Together
I gather the account lead and strategist, focusing on crafting five vital files in straightforward language. It’s about capturing the unspoken knowledge that guides our best decisions.
Step 3: Decide Where the Brain Lives
For solo practitioners, a local folder suffices. Teams, however, benefit from a shared location. Options include Google Drive, Notion, or even a version-controlled system, as long as it serves as a trusted central repository.
Step 4: Set Ownership Rules
I find friction helpful for changes to the soul. The account lead reviews changes, ensuring consistency. Memory changes should be easy for anyone to add, thereby capturing fresh insights on-the-go.
Step 5: Schedule Maintenance
Regular brain maintenance is crucial to prevent rot. Tasks include consolidating duplicates, updating entries, and resolving conflicts. A stale client brain can create more harm than good if left unchecked.
How AI Agents Read the Brain
In practical use, AI tools benefit from having access to the client brain, which helps maintain consistency in outputs and aids collaboration. Whether through comprehensive or selective file loading, the integration should preserve context and avoid unnecessary readjustments.
Version A: Load Everything
A straightforward method is to have AI read all files in the brain folder before starting a task. It incurs some cost, but is often more efficient than repeatedly reexplaining account details.
Version B: Route by Task Type
Selective loading simplifies tasks by having AI access only the necessary files based on the specific task type. It’s a balanced approach many agencies are adopting to optimize efficiency and relevance.
Version C: Vector Retrieval
For agencies managing numerous clients, vector retrieval provides a sophisticated solution. It involves using metadata tags for entries, allowing AI to fetch relevant content effectively while ensuring accuracy and specificity.
Using the Brain Across Different AI Platforms
I ensure the consistency of AI outputs by integrating the client brain into various AI workflows, be it Claude Code or Cowork. The emphasis remains on the AI engaging the soul files at the start of tasks, securing alignment and coherence.
Where This Breaks and How to Fix It
Even a well-maintained client brain can encounter issues like drift or fabrication. Remedy these by ensuring the style guide includes clear examples and that memory entries are frequently reviewed for accuracy and relevancy.
Trust in the client brain derives not from its structure but from its content. A reliable source underpins every memory entry, bolstering confidence and effectiveness.
How to Get Started This Week
To implement this system efficiently, I’ve found it useful to start with a single client, gather the team for a focused session, and conduct a test. This initial phase acts as a proof of concept, validating the utility of a client brain in enhancing SEO tasks.
The real payoff from AI doesn’t come from speed alone but from the context it provides. With a client brain, the gaps usually lost in transition are preserved, ensuring the work is not just faster, but smarter.
I’ve recently discovered that Google has introduced a new feature in Chrome Lighthouse to check for llms.txt files. Though Google mentions that llms.txt isn’t necessary for AI search visibility, Lighthouse has started flagging sites based on their presence.
Google’s latest Lighthouse audits, under the “Agentic Browsing” category, now focus on a site’s usability for machine interaction. I find this interesting as it aligns with Google’s push towards better machine readability.
The new audits are part of Chrome’s evolving “Agentic Browsing” features, which analyze if sites are prepared for automated interaction. This concept came soon after Google issued guidance on AI search optimization, debunking the necessity of llms.txt files in their new guide on generative AI features.
What Lighthouse Evaluates Now. Lighthouse’s Agentic Browsing tests focus on how well my site is built for machine interactions, incorporating various deterministic audits as per Google’s documentation. These checks include:
– WebMCP integration.
– Accessibility tree integrity.
– Layout stability through CLS.
– Presence of an llms.txt file.
These audits help ensure that there’s a machine-readable summary at the site’s domain root. Google explains that without llms.txt, agents might take longer to understand a site’s main structure.
The impact of these audits doesn’t translate into a traditional Lighthouse score but into a fractional pass ratio related to agentic readiness signals.
The Tension. Interestingly, while these audits don’t directly affect SEO rankings, their mention in Google’s readiness checks could make SEOs reconsider their stance on llms.txt files.
Agentic Engine Optimization. Google’s approach aligns with insights shared by Addy Osmani from Google Cloud AI about Agentic Engine Optimization. Osmani emphasizes creating web content that is semantically structured, token-efficient, and easy for AI to process.
SEO vs. llms.txt. According to Google, creating llms.txt or similar files isn’t necessary for AI search success, as outlined in the guide on Mythbusting generative AI search. The AI systems can discover, crawl, and index a variety of file types encountered on the internet.
John Mueller from Google responded to concerns about the role of llms.txt in a discussion with Lily Ray on Bluesky, stating that the use of these files is more for functionality and not directly linked to search engine optimization.
Google’s Take on AI Agents. Besides llms.txt, Google’s Lighthouse guidelines place strong emphasis on accessibility and interface stability. The insight I gained is that AI agents heavily rely on the accessibility tree as their core data model, focusing on integrity and proper layout.
Ultimately, while Google indicates llms.txt isn’t needed for search, including such files might be beneficial for adapting to Google’s evolving tools that prioritize machine readability.
I’m thrilled to share that Google has just unveiled Ask Advisor, a new AI-driven tool designed to transform the way we approach campaign management, analytics, and optimization. Announced at Google Marketing Live 2026, this Gemini-powered AI is here to integrate seamlessly across Google Ads, Google Analytics, Merchant Center, and the Google Marketing Platform.
Making Waves. Ask Advisor is set to be a game-changer, acting as a unifying force that weaves together insights, workflows, and recommendations across Google’s vast marketing ecosystem.
For those of us in marketing, this means we can launch campaigns, analyze performance, and uncover optimization recommendations all without having to juggle between different tools.
Imagine asking Ask Advisor to “find new customers for my hair care products.” It would seamlessly pull details from the Merchant Center and assist in crafting a campaign right in Google Ads.
Understanding the Process. Ask Advisor connects the dots between Google Ads, Analytics, the Merchant Center, and the Marketing Platform via a Gemini-powered interface. This connectivity allows it to access a range of data to create recommendations, automate tasks, and offer insights that align with marketing goals.
It doesn’t stop there. The integration of insights from Google Ads and Google Analytics helps explain campaign performance and suggests subsequent steps.
The aim, Google states, is to democratize advanced campaign management, enabling even those without extensive technical expertise to make the most out of their advertising strategies.
This launch supports Google’s expanding lineup of AI-driven in-product agents, positioning Gemini as a fundamental layer in advertising and measurement tools.
Why This Matters to Us. Ask Advisor symbolizes one of Google’s most direct steps into agent-based advertising workflows.
Instead of interacting manually with separate reporting dashboards, campaign tools, and optimization settings, AI agents are being poised to handle operational tasks and present strategic insights.
The more substantial evolution is structural: Google is anchoring Gemini as the core across its advertising platform, potentially redefining how campaigns are developed, optimized, and evaluated.
Keep an Eye On. The biggest discussion point will be how much control advertisers are willing to cede to AI agents. Transparency over recommendations, automation choices, and reporting accuracy will be under scrutiny as Ask Advisor rolls out.
When You Can Get It. Currently in beta, Ask Advisor is available for English-language accounts, with more features anticipated later this year.
Want to Learn More? Here’s additional news from Google Marketing Live 2026:
I’m excited to share that Google has announced some transformative updates to its search capabilities. These updates include the introduction of information agents and enhanced agentic experiences that will elevate how we interact with search. Google’s AI will continuously scan the web, ensuring we receive the most current information, much like a personal assistant would.
In a recent announcement, Google revealed new search agents, focusing on information agents and additional agentic functionalities within Google Search. These information agents are designed to monitor the web for changes to our tasks, seamlessly supporting us on our journey through various challenges and questions.
Liz Reid, the head of Google Search, stated, “We’re entering the era of Search agents, where you can easily create, customize, and manage multiple AI agents for your many tasks, right in Search.” This new era provides a unique opportunity to tailor search experiences to our specific needs.
Information Agents. These agents are designed to keep us informed about our questions and tasks. Google’s agents will intelligently sift through the internet—exploring blogs, news sites, social posts, and accessing the freshest real-time data on finance, shopping, and sports, to ensure we receive the most relevant updates on our inquiries.
The information agents will then compile an “intelligent, synthesized update” that not only provides the necessary information but also enables us to take action.
The Example. Envision yourself apartment hunting. You can simply input all your specific requirements, and your agent will continuously scan listings, alerting you whenever a match surfaces. Similarly, if you’re keen on not missing any sneaker collaborations from your favorite athletes, your agent will notify you about new releases.
Availability. These exciting capabilities are set to roll out this summer, initially available to Google AI Pro & Ultra subscribers.
Agentic Experiences. Google is also extending its agentic booking capabilities within Google Search to encompass new tasks like finding local experiences and services. Imagine effortlessly booking a private karaoke room for an exact time and with specific food options, all handled by Google Search.
Google will provide the most current pricing and availability information, along with direct links for purchase, streamlining experiences across various services, including home, repair, beauty, and pet care. These features are expected in the U.S. this summer.
Personal Intelligence Expanding. In addition, Google has revealed plans to broaden its Personal Intelligence feature within AI Mode, now reaching around 200 countries and territories, supporting 98 languages.
I’ve seen many technologies come and go throughout my career. I used to chase after every new trend, trying to stay on the cutting edge. However, I quickly learned that this approach often cost me and my clients countless hours, with many technologies fading into obscurity. Does anyone remember Google Authorship?
I’ve realized that by waiting for wider adoption, learning from early adopters’ mistakes, and catching up quickly, I avoid wasting time and create more value. This approach has been invaluable to me.
However, some moments in technological advancement stand out—when being an early mover means not just succeeding but helping shape the future. The first people to realize the importance of PageRank and started building links can relate. WebMCP feels like another one of those pivotal moments, only larger.
The change we’re facing isn’t just about search engine mechanics or generative engine visibility. Discovery itself is evolving, and the entities performing this discovery are changing too.
I remember the age-old debate in SEO circles—should we focus on search engines or people? My answer is both. Yet now, this paradigm is shifting. What happens when discovery shifts from human-driven to being guided by AI agents?
When you ask ChatGPT a question today, it processes information, conducts additional searches, asks follow-ups, and delivers conclusions. The AI agent plans and decides for you, influenced entirely by its data sources and interpretive frameworks.
This evolution represents just one chapter in the ongoing story of discovery:
Discovery v1: Experiential interactions and word of mouth dominated.
Discovery v2: The written word took prominence in libraries and print media.
Discovery v3: The web spawned directories and search engines.
Discovery v4: Today, we see AI and LLMs increasingly aid discovery.
Discovery v5 (coming soon): Agentic systems will advance to perform actions autonomously.
Embracing Discovery v5 could offer us significant liberation—freeing our minds from mundane decisions, and enabling a focus on what truly matters.
The path to Trustable AI is underway. I now trust AI systems with everyday queries, relying on them more each time they enhance their capabilities.
Would I trust an AI to handle complex tax or health questions? Not entirely. Would I ask it to help plan dinner or schedule my day? Definitely.
This gradual trust expansion parallels past experiences with technology. As it grows, so does our reliance on agents to act on our behalf.
The tangible impact is visible: Automating grocery reorders or offering extraordinary travel deals are low-risk, high-reward changes.
The skepticism towards relinquishing control to technology is as old as technology itself. From fear of entering credit card details online to today’s reliance on smartphones and GPS, each shift was gradual but unstoppable.
WebMCP, which facilitates AI interaction with websites, is a browser-native web standard. It’s gaining momentum, authored by Google and Microsoft. It’s about easing AI’s job in understanding actions on websites, not replacing human interaction.
AI doesn’t need to infer tasks. WebMCP allows clear communication of a site’s capabilities, marking a shift like early schema markup days.
Engaging with this framework ensures your site is AI-ready, simplifying AI interaction.
WebMCP impacts discovery, influencing which sites AI agents prefer. Having your site AI-visible can make or break engagement in the emerging landscape of Discovery v5.
I’m taking advantage of this moment, despite my usual skepticism of early adoption—it feels different this time.
I’ve noticed it’s not uncommon to come across articles proclaiming that AI agents are about to revolutionize Google Ads, SEO, or social media. Initially, these AI agents seem promising, at least in theory.
But when I dive deeper into what data these agents actually utilize, it’s almost always platform-native. For Google Ads, this translates to impressions, clicks, conversions, and ROAS.
This simplistic approach is why PPC AI agents often stumble right from the start. If they only have platform-specific data, managing true marketing strategies becomes impossible.
Why Many PPC Agents Are Just AI Assistants
Many tools labeled as PPC agents are mostly AI assistants, focusing on tasks such as:
Generating various headline options
Describing product images for Responsive Search Ads
Drafting CTAs for Performance Max asset groups
While these tasks are beneficial in freeing up time, they’re not quite the PPC agents they claim to be—they’re just dressed up generative AI tools.
A true PPC agent operates directly on an ad account by analyzing performance data and making strategic decisions, like adjusting budgets and optimizing campaign structures based on informed insights.
How AI Agents Create a Closed Loop
Google Ads has a limited view of your business data, causing AI agents to often optimize a closed loop focused solely on improving platform metrics, which may negatively affect business performance.
For instance, Google Ads doesn’t know specifics like average deal size or which products have high margins. This ignorance can lead to suboptimal decisions.
Performance Max: A Precursor to AI Challenges
This conundrum isn’t new. PMax campaigns already demonstrated the pitfalls without adequate data, as they often optimized towards the wrong goals without necessary business insights.
PPC Agents Risk Misalignment Without Business Data
AI agents exacerbate the speed at which misaligned strategies can cause harm. Even the best systems need backend business data to make informed decisions, just as your agent would.
3 Essential Types of Business Data for PPC AI Agents
To enhance PPC agent performance, integrating CRM, product, and operational data is crucial.
1. CRM Data
CRM data is vital for understanding lead values beyond mere conversion counts. You can bridge this gap with offline conversion tracking or direct CRM access for a deeper analysis.
2. Product Margin Data
Understanding product margins is essential for eCommerce success. This data should come from supplementary feeds or direct backend connections, allowing for more strategic budget allocations.
3. Operational Data
Operational signals, like fulfillment capacity, also impact decision-making. Effective coordination and data flow help prevent suboptimal choices that might appear beneficial only theoretically.
Questions to Ask Before Building a PPC AI Agent
Before developing a PPC AI agent, pinpoint the essential business data required to optimize campaign performance, starting with OCT and progressing to direct CRM links for comprehensive insights.
Ultimately, the challenge isn’t building the agent but integrating it seamlessly with business realities for genuine value extraction.
I’ve often heard from paid search managers that dealing with AI agents can feel repetitive. Imagine exporting your performance data, pasting it into a chat window, receiving a useful answer, and then having to repeat the process every day. That doesn’t sound like automation, does it? It’s just good old manual work with a tech twist.
Interestingly, the issue isn’t with the AI tools themselves. Many of them excel in data analysis when they have access to the right information. The real hurdle is providing this data to them in real time, without constantly needing a human to copy it over. This data wall explains why many PPC accounts today operate nearly the same way as they did before the advent of AI agents.
Every ad platform tends to operate in isolation. Google Ads might record conversions, while your CRM notes whether those leads are qualified, and your inventory system checks stock availability. Without deliberate integration, they each function in their own silo. PPC managers have traditionally bridged this gap manually with regular exports and cross-referenced spreadsheets. Although this worked while humans managed it, it doesn’t hold up when an AI agent needs to take action in real time.
Consider a keyword with good volume and a satisfactory CPA, according to Google Ads. But in HubSpot, these could be marked as disqualified leads. The AI, lacking this context, continues its work blissfully unaware, leading to unnecessary budget spend until someone catches the discrepancy during the monthly review. This is a data access problem that better prompts alone can’t fix; a robust data pipeline is essential.
The Model Context Protocol (MCP) is here to address this by providing a standardized way for AI clients to connect to various data sources. Before MCP, one would need to build separate connectors for systems like Google Ads, CRMs, and inventory systems, but MCP simplifies this connection significantly.
Now, with MCP, an AI agent could efficiently work with Google Ads and CRMs like HubSpot, cross-referencing conversions with CRM dispositions. This setup can automatically adjust bids based on data, eliminating the need for human intervention in the reporting process, saving valuable time.
Yet, having an open pathway to data without safeguards introduces new risks. Imagine an AI with write access to a Google Ads account. Without defined parameters or constraints, actions taken by the AI could become unpredictable. This unpredictability is why guardrails must be established around the AI, rather than relying on the AI tool itself to handle this responsibility.
Optmyzr’s MCP allows advertisers to control what actions the AI can take, ensuring a balanced approach to AI management. This ensures the AI can effectively manage campaigns while staying within safe operational parameters.
The MCP from Optmyzr integrates these controls into its system, allowing AI agents to perform complex tasks such as executing a full Rule Engine strategy from a simple directive while ensuring the appropriate checks and balances are in place. The result is an agent capable of operating with the precision of a seasoned PPC strategist across your entire portfolio, offering a level of intelligence and safety unattainable through raw API access alone.
For those who wish to explore the possibilities of AI with care, Optmyzr’s MCP provides a secure and efficient pathway, integrating seamlessly with tools like Claude Desktop or ChatGPT for a comprehensive AI-powered approach to managing marketing campaigns effectively.
When it comes to Google Ads management, I’ve always followed the same routine: logging in, evaluating the performance, making updates, and crossing my fingers for success.
Despite advances in technology, from spreadsheets to automated bidding, the fundamental process hasn’t changed—until now.
Today, groas is shaking things up with a new, fully autonomous model for managing campaigns. The aim? To seamlessly handle the entire advertising process without constant manual input.
This revolutionary system has been in the making for years. Our company has developed an AI-driven approach that runs 24/7, matching or even exceeding industry benchmarks in PPC performance.
From building a campaign to managing bids, creating ad copy, and expanding keywords, this AI network takes care of everything autonomously.
When we first launched groas as a lightweight platform, it primarily provided optimization tips. But the true game-changer came from real-world data.
Early adopters joined from various industries, providing invaluable data that shaped groas into the powerhouse it is today.
Thanks to this diverse data from real campaigns, our AI has become skilled at understanding what truly works.
Our founder, David Pourquery, once shared the frustration of valuable recommendations sitting idle, awaiting approval. Now, our system makes those changes automatically.
We recently overhauled our system, creating interconnected AI agents that process mountains of data every hour, lifting the limits of manual management.
Ads management tasks are automated, allowing human professionals to focus on bigger strategic goals. groas delivers dynamic landing pages through a single JavaScript line, enhancing conversion rates continuously with A/B testing.
I don’t have to check in daily. Weekly reports summarize the autonomous progress while a human PPC manager supervises it all.
Starting off with groas is quick and easy. My personal account manager handles the setup, providing a detailed action plan within a day.
groas now autonomously manages significant monthly ad spends, all through word-of-mouth and direct referrals—without a dime spent on advertising.
Our client base includes businesses seeking consistent results and agencies leveraging groas for streamlined campaign execution.
With Google’s lean towards automated ads, groas offers a unique, fully autonomous solution that maintains strategic involvement through a dedicated manager.
The industry has long debated automation degrees in PPC. groas answers by fully automating while managing extensive ad spend.
groas has transcended traditional approaches; we’ve reduced the need for recommendation engines entirely.
Our services start at $999 per month, scaling as needed. This model requires a minimum $2,000 monthly ad spend to optimize data effectively.
Diving into the world of technical SEO for generative search has had me rethinking how AI agents interact with my site. It’s not just about indexing anymore; it’s about how AI systems generate answers. My focus is now on ensuring AI agents can access and interpret my content smoothly, enhancing the chances that I’ll be cited in AI-generated responses.
When I consider generative engine optimization (GEO), I’ve realized that while the underlying tools and frameworks aren’t new, the way I implement them makes the difference in my content being surfaced or missed.
It means paying close attention to how AI agents access my site, structuring my content for easy extraction, and ensuring it can be reliably interpreted and reused in AI-generated responses. This is about precision and strategic structuring.
Agentic Access Control: Managing the Bot Frontier
Using robots.txt strategically has become vital. It’s essential for me to specify which crawlers can access what parts of my site. For instance, I might decide that a training model like GPTBot should access my /public/ folder but keep my /private/ folder off-limits, implementing it as follows:
The choice between model training and real-time search is crucial. Often, I find myself balancing whether to disallow GPTBot or allow OAI-SearchBot. Considering Perplexity and Claude standards within my robots.txt is another layer I need to manage:
Claude
ClaudeBot (Training)
Claude-User (Retrieval/Search)
Claude-SearchBot
Perplexity
PerplexityBot (Crawler)
Perplexity-User (Searcher)
I’ve also had to integrate the new protocol, llms.txt. Although not universally adopted, it’s a structure I find useful for guiding AI agents in understanding my content better. If you’re interested in following Perplexity’s llms.txt, you can explore it here:
llms.txt: A concise map of links.
llms-full.txt: An aggregate of text content that allows agents to bypass crawling my entire site.
Even if Google and others aren’t reading llms.txt right now, I believe it’s worth preparing for future needs. John Mueller has shared insights on this which you can read here.
Extractability: Making Content ‘Fragment-Ready’
In the realm of GEO, I’ve been focusing on creating content fragments because AI systems value precise and concise information. This means avoiding bloated content that can hinder AI retrieval due to issues like:
Challenges with JavaScript execution.
Overreliance on keyword optimization instead of entity optimization.
Poor content structures lacking clear answers.
To make my core content visible and accessible to various AI entities, semantic HTML components like <article>, <section>, and <aside> have become essential tools. This separation helps the essential facts stand out, feeding search engines and AI bots effectively.
Technical SEO is evolving, and as I adapt, I’m focusing not just on visibility, but on becoming a source of truth for the world’s AI models. By using structured data efficiently, implementing robust access control via robots.txt, and refining my content’s extractability, I’m setting the stage for success now and into the future.
Ensuring my strategies are working requires thorough auditing. I focus on areas like citation share, log file analysis, and zero-click referrals to measure how effectively my content is influencing the AI-driven world. This helps validate my efforts and enhance KPIs.
Scaling GEO into 2027
Looking ahead to 2027, I’m prioritizing automation to minimize manual optimization work. The goal is to leverage every SEO tool available, ensuring my site is a robust source of truth amid AI advancements. Starting with basics like robots.txt and moving towards more sophisticated structures, my ongoing goal is to scale efficiently and effectively.