When I think about Google’s journey with artificial intelligence, Sergey Brin’s admission strikes a chord. He candidly revealed that Google ‘for sure messed up’ by not prioritizing AI investments at the right time. Looking back, it seems clear that we released groundbreaking research but didn’t capitalize on it to ride the current wave of generative AI.
Google’s cautious approach always intrigued me. According to Brin, there was hesitation and fear about potential missteps, especially since chatbots could ‘say dumb things.’ It’s fascinating to hear him acknowledge that Google didn’t move quickly enough after publishing the Transformer paper.
The hesitation seemed to stifle opportunities. Brin admitted that while Google was reluctant, companies like OpenAI leapt forward with brilliant foresight. They seized the moment, leveraging insights and even talent, such as Ilya Sutskever, to drive AI innovation.
Reflecting on the past, Brin shared, “In some ways, we for sure messed up in that we underinvested… eight years ago when we published the transformer paper. We didn’t take it all that seriously and didn’t necessarily invest in scaling the compute. And also we were too scared to bring it to people because chatbots say dumb things. And you know, OpenAI ran with it, good for them.” He graciously acknowledged the value of the history we’ve built.
The current landscape still favors Google, as Brin points out. Years of AI research and development, deep learning, and our robust infrastructure continue to provide a competitive edge. This bedrock underlines the control we maintain over key technologies driving AI today.
Why does this matter? Brin’s insights shed light on why Google’s AI-driven changes in search sometimes seem sudden and erratic. Our earlier caution means we now find ourselves speeding ahead, possibly too quickly, to bridge the gap. The fluctuating nature of Google Search is a byproduct of this rapid adjustment process.
When Brin spoke about AI’s future, he characterized the field as highly competitive and ever-evolving. He mentioned, “If you skip AI news for a month, you’re way behind.” It’s uncertain what the ultimate potential of AI is, or if there’s a limit to its intelligence.
On a personal note, Brin shared that he often uses Gemini Live for engaging conversations while driving. Interestingly, he noted that the public version is outdated, with a “way better version” on the horizon.
Looking back and forward, Brin’s remarks at Stanford were part of an event celebrating the School of Engineering’s century-long legacy. The discussion touched on Google’s early days, its culture of innovation, and the present AI ecosystem. You can watch the full video here.
As I dive into the evolving world of SEO, I’ve noticed one thing: the industry is entering its most unpredictable phase yet. With traffic on the decline and AI increasingly handling informational queries, it’s clear that the landscape is shifting beneath our feet.
It’s fascinating to observe how social platforms are now serving as search engines, and Google is transforming from a gateway to a comprehensive answer engine. This transformation leaves many of us in the industry uncertain about what metrics matter, what we should optimize, and essentially, what SEO’s role truly is in this new digital era.
Despite the chaos, I’ve found clarity in one specific marketing metric that cuts through the noise: share of search. This metric offers a straightforward insight into brand health and potential future demand, aligning marketers and SEOs with confidence.
Share of search becomes particularly important as we notice a significant shift in how discovery and measurement need to adapt. The days of accidental discovery through traditional search behavior are dwindling.
AI and platforms like Meta are increasingly providing direct answers without directing traffic elsewhere, shifting the focus towards metrics that provide a clearer indication of consumer interest, like share of search.
Interestingly, share of search, a concept developed by James Hankins and Les Binet, calculates a brand’s search volume against the total search volume for its category. This simple yet powerful metric correlates strongly with market share and future buying behavior.
In our rapidly changing environment, share of search provides a critical signal for marketers, showing whether a brand is being searched for more or less compared to competitors. This insight offers a palpable reflection of underlying consumer interest and demand.
While traffic as a metric is losing its significance because of AI pre-answering queries, share of search cannot be manipulated easily. It stands resilient as a reflection of authentic consumer desire.
Moreover, this metric crosses platforms effortlessly, as people now search across various digital spaces such as Amazon, TikTok, YouTube, and potentially even LinkedIn. Share of search adapts to fragmented discovering behavior precisely.
It’s exciting to see how, even if AI-driven systems like ChatGPT rarely generate clicks, they often trigger brand searches, emphasizing the importance of this metric as a measure of marketing effectiveness.
For SEOs like me, adopting share of search means transforming our roles from content producers into strategic partners, providing deeper insights into consumer behavior and brand demand.
Ultimately, embracing share of search elevates our value within an organization, offering a fresh narrative around brand visibility and performance. As AI continues to reshape the digital landscape, this metric is becoming indispensable for those of us in SEO and marketing. I encourage everyone to learn more about this compelling metric and explore its potential to transform how we measure success in the AI era.
I recently discovered that uncontested ads might be silently eating away at my holiday budget. Even when I’m the sole bidder, my CPCs remain stubbornly high. Here’s how I began to reclaim those wasted dollars.
This holiday season, Google Search and Shopping Ads are projected to surpass a staggering $70 billion in spending. However, many advertisers, myself included, overlook a critical flaw in Google’s auction system that drains our funds—even in the absence of competitors.
The team at BrandPilot identifies this issue as the “Uncontested Google Ads Problem,” a significant yet often ignored source of wasted ad spend during peak times.
During SMX Next, I learned from John Beresford, the Chief Revenue Officer at BrandPilot, about a little-known quirk in Google’s auction logic. It’s fascinating how this can lead advertisers like me to overspend on our brand terms, shopping placements, and category keywords because Google doesn’t automatically lower our CPCs when no one else is bidding.
Instead of enjoying lower costs as the sole bidder, I found myself paying the same high rate as if competitors were still active. It’s a situation that unfolds thousands of times a day for major brands, and like me, many marketers don’t even realize it.
In John’s session, we explored:
Understanding why “competition gaps” are far more frequent than we think.
Discovering how uncontested moments can warp CPCs, even on brand keywords.
The potential of real-time auction visibility—and how AI is revolutionizing the field.
He also shared how advertisers are deftly reclaiming wasted spending and channeling it back into growth, without giving up impression share, traffic, or revenue.
Identify why CPCs are artificially high when competitors are missing.
Calculate the true financial impact of the Uncontested Ads Problem on your budget.
Execute AI-driven bidding and suppression strategies to avoid self-bidding and increase ROAS.
If you’re managing Google Search or Shopping campaigns this holiday season, this session is a must-see. Learn how to keep Google from sneaking off with your budget and start converting those savings into real performance improvements.
On December 11th, I was excited to learn about OpenAI’s announcement of GPT-5.2, their most advanced frontier model yet. Knowing how transformative AI can be, I was thrilled to see that Profound is now tracking GPT-5.2 responses in ChatGPT. This upgrade is integrated across our entire product suite, including Answer Engine Insights, Prompt Volumes, and Agent Analytics.
Starting today, every ChatGPT response on my dashboards reflects this cutting-edge model. It’s exhilarating to think about the enhanced capabilities and strategic insights that I can now access thanks to this update.
I recently discovered that Google’s AI made several damaging claims about me, alleging that I had been suspended for selling sick notes. As a UK doctor and YouTuber, I was stunned to find out that these false allegations could severely impact my career.
In a recent video, I explained how Google’s AI created a complete fiction about my professional life, despite my spotless record over the past decade. This alarming revelation raises questions about defamation and the responsibility of AI-generated content.
Why this matters. When AI-generated narratives start presenting false and damaging claims as facts, the implications for defamation and accountability are significant. It’s crucial to understand whether these AI statements are shielded by Section 230 protections.
What Google’s AI claimed: The AI falsely stated that:
My suspension by the medical council occurred in mid-2025.
I profited from selling sick notes.
I exploited patients for personal gain.
My online fame led to professional discipline.
‘None of this is true.’ With nearly 500,000 followers, I have no idea how many people might have seen this false information before I discovered it. Replicating this erroneous overview revealed even more fabrications, including misleading insurers and content theft.
How did this happen? I believe that Google’s AI pieced together a narrative from random bits of data, wrongly associating my identity with unrelated events:
I hadn’t uploaded to my YouTube channel, “Sick Notes,” in months.
Another doctor, Dr. Asif Munaf, was tangled in an actual sick-note scandal.
Why this is more than just a mistake. The AI didn’t question its fabricated claims; it presented them as undeniable truths. This poses issues because:
AI responses are often viewed as authoritative.
Users cannot discern sources, biases, or motivations.
There’s no straightforward process for correction or accountability.
The allegations targeted me, a private individual, outside of public controversy.
The big legal question. Is Google’s AI guilty of defamation? Is it protected under Section 230? Ultimately, courts may decide. However, some legal experts suggest:
AI outputs aren’t third-party speech.
They constitute new, published statements.
False facts could indeed be defamatory.
Resolved? Searching for my supposed suspension now returns a different, unclear explanation:
“Dr. Ed Hope (of the ‘Dr. Hope’s Sick Notes’ YouTube channel) faced scrutiny and suspension…” This narrative erroneously conflates events and imagines contexts.
A more recent search presents even vaguer details, blending potential fictional characters with real drama.
For the last two years, I’ve been swept up in the AI gold rush era. It’s reminiscent of what Taylor Swift would call the “Lover” phase—everything was shiny, fresh, and filled with potential.
My approach? I tried to buy it all.
But now, I’m shifting gears to a “Reputation” phase, which feels darker, edgier, and all about the receipts.
Noticing headlines like Microsoft’s decision to lower AI sales targets got me thinking. People framed it as a disappointment, but what I see is a market maturing.
As we’re evolving, I’m realizing that we’re leaving behind the AI gold rush era. Microsoft’s recalibration is just one sign that we’re stepping into AI’s Production Phase era.
Conversations are changing: I’m more focused on whether these tools actually work within my business, connect to our stack, and drive revenue.
There’s a shift happening as the AI market remains a bit unstable. With almost 40% of U.S. consumers having tried generative AI, regular use isn’t quite there yet, as shown by moves in platform loyalty.
This instability means that for me, orchestration is key to staying future-proof in a fragmented ecosystem.
The martech scene has exploded with over 15,384 solutions available, yet I see only 33% of tech being fully utilized. We were paying for a full suite, but truly benefiting from just a third of it.
During the rush, we bought point solutions to address specific problems, but lacked a conductor to bring everything together harmoniously.
This results in what I’d call Pilot Theater—demos that impress but fail to deliver ROI because they’re trapped in isolated silos.
Imagine your P&L hit by these issues: budget disconnects, experience breaks, and content gaps. These gaps are a signal, but what’s missing is coordination, and the pressure is mounting with CEOs keen for AI ROI.
Moving forward, I have to go beyond automation, to embrace agentic orchestration—this is where systems don’t just automate, but adapt and integrate.
Orchestration becomes the nervous system of my marketing operations. It’s my survival strategy in a rapidly evolving AI space.
Real orchestration happens now, with intelligent feedback loops replacing manual processes. Here’s how it’s working for me:
I’ve seen how orchestration aligns efforts, such as in budget fluidity, buying group alignment, and closing content loops to meet real buyer needs.
As a leader, I’m now part of what’s known as the “Builder” generation. Marketing teams, including mine, are becoming more like product teams, building custom platforms to meet our unique needs.
Integration is key, and it’s becoming clear: Orchestrators are now the leaders. This isn’t the end of AI, but the end of tourist AI. Growth now requires intelligence, not volume.
My advantage lies in developing an AI nervous system that is effective across channels, capitalizing on opportunities before they slip away. The orchestration era in AI is here to stay and it’s time for orchestrators, like myself, to lead.
Every year, Black Friday offers a unique glimpse into how consumers search, compare, and decide. This year, it added another layer: it became a real-world arena to see how AI models comprehend commerce amidst genuine demand.
I embarked on a journey to test major large language models (LLMs), analyzing 10,000 responses to understand how these systems perceive the retail landscape and the signals that shape their responses.
As I dissected the dataset, a pattern was unmistakable: Black Friday acts as a genuine stress test for AI-driven discovery.
The sheer number of queries and the diversity of categories reveal the sources, structures, and behaviors LLMs rely on for reasoning about products, retailers, and consumer intent.
The outcomes offer a sneak peek into how AI search is transforming—and how this will impact the broader commerce ecosystem.
TLDR; LLMs lean heavily on a limited range of external domains with YouTube, large retailers, and U.S. review media leading the charge.
Generalist retailers dominantly capture nearly half of all retail citations, serving as the recurring funnel LLMs use to address shopping queries.
Social and user-generated content see an 8.1% surge during Black Friday, as conventional retail and media sites experience a decline.
Off-page signals like Reddit, YouTube, Amazon, and Consumer Reports are vital, equally important as on-page content for shaping LLM comparisons and recommendations.
Structured comparison content wields significant influence, far surpassing branded assets.
The behavior of LLMs differs not only from Google but also from each other, with each platform like Gemini, OpenAI, and Perplexity offering unique formats, lengths, and reasoning patterns.
Unlike traditional search, where the process begins with a query leading to a list of ranked results, AI search reverses this. It starts with a model’s internal web of relationships, sources, and signals to construct a response.
In our review of the top 50 most-cited domains across 10,000 LLM responses—all centered around deals, reviews, and product recommendations—the distribution was notably skewed:
YouTube led with 1,509 citations, followed by Best Buy with 950, Walmart with 885, Target with 477, TechRadar with 355, RTings with 342, and Consumer Reports with 325.
This cluster shapes much of the commercial “knowledge” from which LLMs draw. It gravitates towards large retailers, global media outlets, and platforms specializing in comparisons and reviews.
In analyzing 10,000 responses, I compared the week leading up to Black Friday with the event itself. Pre-Black Friday, responses reins focused on planning behavior.
Retail and brand domains: 59.6%
Media: 23.4%
Social and user-generated content: 17%
When Black Friday commenced, the mix rapidly evolved. Social and UGC content jumped to 25.1%, gaining significant share, while retail and media slightly retreated.
This shift within the models mirrors consumer behavior but also highlights the models’ reliance on conversation-driven content for in-the-moment decision cues.
One of the most transparent insights is the weight third-party domains carry on AI reasoning. Today’s LLMs thrive by absorbing as much human interest in products as possible. Huge volumes of consumer insights, reviews, product demos, sentiment, and structured data guide how models reason and decide.
An analysis revealed key off-page signals LLMs depend on:
Reddit: 34%
YouTube: 19.5%
Amazon: 15.5%
Business Insider: 9.2%
Walmart: 8.9%
Each domain influences different aspects of the model’s decision-making. Across the board, LLMs lean on content that captures human interest, organizes consumer options, and mitigates uncertainty through verifiable data.
While third-party domains reign supreme, brand websites still hold measurable sway. They are vital for any consumer brand aiming to excel in AI discovery.
A site’s architecture plays a crucial role in how a model interprets a brand. Homepages account for 40% and serve as the primary identity layer—establishing tone, positioning, and offering quick semantic signals to models.
Blogs and product pages clarify brand definitions and long-tail context, providing the factual details models need.
Brands that rely too heavily on promotional copy, weak hierarchies, or thin product content risk sacrificing major visibility.
Across the entire dataset, certain retailer categories led the charge in model responses.
Generalist retailers hold 48% of the conversation. Walmart, Target, and Best Buy capture almost half of all retail citations. Their range, familiarity, and content depth make them central figures in LLM commerce reasoning.
Electronics specialists grasp 23% of the share. Best Buy leads, trailed by Newegg and Micro Center, with tech-focused queries often directing models toward these sources.
Other verticals lag behind. Despite strong category leaders, sectors like fashion, beauty, and home capture smaller portions due to the content volume disparity compared with generalist retailers.
Reviewing the platforms uncovered another pattern: major LLMs not only offer different answers but exhibit distinct thinking styles. Each platform has its own rhythm, structures, and styles for presenting commercial information.
Gemini provides the most detailed responses, with essays averaging 606 words, using lists and headings extensively.
OpenAI stands in the middle, averaging 401 words per response, with high list usage and balanced headings.
Perplexity shifts towards brevity with an average of 288 words, favoring short summaries akin to executive briefs.
These differences define unique retrieval and reasoning methods, shaping how each platform interprets brands, categories, and commercial intent.
The data presents a clear direction: AI search is forging its ecosystem, driven by familiar SEO inputs, source quality, content structure, and off-page signals, all interpreted to deliver precise answers.
If your content isn’t well-structured and present across the web, it risks becoming invisible to AI platforms delivering answers or product suggestions.
As this new environment evolves, it’s crucial for retailers and brands to rethink their communication strategies across the entire digital landscape.
On-page actions that matter:
Develop semantically coherent homepages that convey the brand, product categories, and relevance to core queries. LLMs prioritize clarity over cleverness.
Strengthen product pages with factual content, clear specifications, and Q&A sections aligned with user research intents.
Establish educational content clusters tied to core product themes, serving as reusable frameworks for AI models.
Off-page actions that matter:
Foster comprehensive review ecosystems and discussion forums to validate trust signals LLMs recognize with product quality.
Ensure visibility in media driven by comparisons and recommendations. Regularly appear in “best of” lists, product roundups, and influencer content.
Invest in rich media showcasing product value, particularly on YouTube and TikTok. Video content helps train LLMs on product use cases, reflecting sentiment, and experiential value.
Maintain accurate, indexable product data in marketplaces like Amazon, Walmart, and Etsy to enhance AI discovery pathways.
OpenAI’s Shopping Research announcement escalates the stakes. With ChatGPT, OpenAI tracks real-time consumer research behavior, turning preferences into a user-trained targeting engine for commerce.
This isn’t just AI learning about your product. It’s AI absorbing consumer shopping behavior, revolutionizing discovery through an active AI participation model.
Brands not infused into these AI systems risk invisibility during AI-driven consumer journeys.
What Black Friday revealed was more than top-selling products; it showed how LLMs operate under real demand, revealing their reasoning, referencing, and prioritizing patterns.
The advent of AI-native visibility requires structured, semantically rich content, adequately represented across the right off-page ecosystems to align with major AI models’ reasoning.
Black Friday might be the stress test, but the real transformation is only just beginning.
In my quest to find the top HVAC SEO agencies of 2025, I dove into our comprehensive research findings. Our team evaluated over 40 agencies, narrowing it down to the top 9, based on these key criteria:
Past Clients (25%): I found that an agency’s previous work with HVAC companies is crucial. A robust portfolio of client successes speaks volumes about their ability to design effective SEO campaigns.
Year Founded & Median Employee Tenure (10%): SEO has evolved dramatically, so agencies with a track record of adapting to Google’s frequent algorithm updates stand out. Longer employee tenures often indicate a culture of continuous learning.
Founder Status & Leadership Experience (20%): Leadership plays a pivotal role in campaign outcomes. I rated agencies based on leadership expertise in HVAC marketing, favoring those still led by their founders.
Average Reviews (20%): I examined client reviews on various platforms, prioritizing those from HVAC businesses for greater credibility.
Specialty (10%): Agencies with a niche focus on HVAC and proficient in SEO were favored in our analysis.
Media References (5%): Although a smaller factor, being frequently cited by reputable media indicates an agency’s visibility and authority.
GEO Offering (10%): I considered agencies that expanded their services to include generative engine optimization, helping HVAC companies rank in AI-powered search environments like ChatGPT.
Here are my findings on the top HVAC SEO agencies:
The Top HVAC SEO Agencies
First Page Sage: An industry leader, renowned for SEO innovation. Founded in 2009, they lead the charge with thought leadership-based SEO. Their notable clients include Windy City Ventures and Four Seasons Heating & Plumbing.
Lemon Seed: Established in 2019, this agency covers the full marketing spectrum for HVAC companies, with a flair for brand design. Their work with Krueger and Climate Plus sets a high standard.
Mediagistic: This agency blends traditional marketing with SEO, serving large enterprises since 1999. They excel in media buying across diverse channels.
Marketing Eye: Founded in 2004, they focus primarily on technical SEO, optimizing existing web content, ideal for firms with robust in-house marketing teams.
LocaliQ: Known for geotargeted SEO, they specialize in boosting local visibility for small businesses. Founded in 2004, they offer flexible scaling as businesses grow.
Scorpion: Providing comprehensive digital solutions since 2001, Scorpion is a go-to for businesses looking for seamless marketing integration.
HVAC Webmasters: True to their name, they focus on all things digital for HVAC firms. They excel in local SEO and web design, being a great match for companies needing a digital facelift.
Metric Theory: Since 2012, they combine PPC and SEO, helping businesses bridge gaps with quick lead generation in mind.
Today, I am thrilled to share that Search Engine Land is celebrating its 19th anniversary!
Nineteen years is an incredible milestone. For almost two decades, we have been diving deep into the ever-evolving world of search engines, always striving to make sense of the changes and challenges Google and the search industry present.
This year, 2025, has been one of the most transformative since our launch in 2006. The rapid pace of change has been exhilarating.
Through it all, our mission remains steadfast: to provide clear news, insightful analysis, and practical guidance to help you navigate the world of search.
Before we look to the future, I want to express my heartfelt thanks for your support and reflect on the past year with you.
Thank you for reading
Sincerely, thank you for being with us.
Every day, we focus on you: what you need to know, what really matters, and what changes will impact your work today or your strategy months down the line.
Our goals include:
Focusing on meaningful stories, not filler.
Delivering news clearly and quickly.
Providing essential context and expertise.
Being a dependable resource in a fast-changing industry.
Helping you anticipate where search is heading, even when it’s unclear.
If you haven’t yet, I encourage you to subscribe to our daily newsletter for a curated summary of all things search, helping you stay updated without feeling overwhelmed.
Thank you to the Search Engine Land team
Our team’s passion is what has driven our success for almost two decades.
Though small, our team accomplishes significant and impactful work because we are mission-driven and dedicated to search.
I extend my greatest thanks to:
Barry Schwartz. With 22 years of experience, Barry’s passion for search ensures complex topics become understandable. He is indispensable.
Anu Adegbola. Focusing on paid media, Anu offers clarity amidst constant changes with her insightful writing.
Angel Niñofranco. Angel plays a crucial role in our SME articles through his coordination and editorial oversight.
Kathy Bushman. Kathy’s behind-the-scenes expertise ensures SMX events are seamless and valuable.
And to the entire team at Third Door Media within Semrush, whether or not your name appears here, your contributions are invaluable.
Top highlights from the past year
Despite the uncertainties of this year, Search Engine Land thrived, thanks to the trust of our community.
SMX Advanced returned in person for the first time in 6 years
This was arguably the highlight of the year. SMX Advanced’s return in person after six years was electrifying.
With attendance surpassing expectations, the sessions were dynamic, and conversations felt like reunions for the search marketing community. It was clear that we all missed these face-to-face exchanges about AI, Google’s updates, and more.
We learned again that when great minds gather, extraordinary things happen. We eagerly await our next gathering in Boston, June 3-5.
Defining industry coverage of AI Overviews and the new era of search
This year, more than ever, transformed the search landscape. We’ve provided the clarity and reporting needed in this evolving environment.
Our readers rely on us for insights during times of change, and we take pride in shaping the industry’s future understanding of search.
Subject Matter Expert (SME) program growth
This year saw a surge of new and returning readers turning to us for insight into SEO and PPC shifts, from AI to SERP experiments.
Our growth owes much to our fantastic contributors, and I extend my gratitude for their impactful work.
Looking ahead: What’s next for Search Engine Land
As we embrace our 19th year, our resolution is steadfast: to offer unparalleled coverage of search-related topics.
This year, you can anticipate:
Continued breaking news on SEO, PPC, AI, and more.
In-depth analysis, guides, and contextual explainers on industry evolution.
SMX events tailored around the nuances of AI-driven search.
Enhanced expert viewpoints, data, and market clarity.
Mark your calendars for:
SMX Advanced: June 3-5
SMX Next: Nov. 18-19
We have much in store for you, with the aim of equipping you with the insights necessary for your best work.
A brief look back to where it all began
Launched on Dec. 11, 2006, Search Engine Land began with a vision of search as a vast community. A place of exploration, connection, and evolution. Over these years, it’s grown beyond our expectations.
The mission remains the same:
Search Engine Land is your destination to remain informed, educated, and connected within the world of modern search engines.
Thank you for 19 incredible years
From everyone here at Search Engine Land and Semrush, thank you for your readership, engagement, and passion for the evolving world of search.
Here’s to a promising rest of 2025 and a remarkable 2026.
See how collaborating with LLMs can transform your content by converting customer, expert, and competitor data into actionable insights.
When I think about large language models (LLMs), one major discussion point is their ability to scale content creation. It’s a tool we’re all tempted to lean on heavily. However, balancing efficiency with creativity is key.
With our busy schedules, boosting productivity is essential. Imagine using tools like Claude and ChatGPT not just for speeding up processes, but also for adding a personal touch to your website and making your day-to-day tasks easier, all without sacrificing creativity.
This journey explores how to:
Analyze customer feedback and questions comprehensively.
Streamline the gathering of detailed insights from subject matter experts.
Conduct competitive analysis effectively.
These tasks, often done manually, can be remarkably enhanced with automation, giving you an edge by rooting your approach in customer and market realities instead of working in a vacuum.
By tapping into this information, I can better connect with my audience, avoiding the pitfalls of an echo chamber.
Analyzing Customer Feedback at Scale
One outstanding feature of LLMs is their scalability in processing data, identifying patterns, and uncovering trends—tasks that might otherwise take me or a colleague days or even weeks to complete.
If you’re not part of a global enterprise with a dedicated data team, LLMs are your next best ally to substitute those capabilities. Focusing on customer feedback, for instance, could mean the difference between success and redundancy. The thought of sifting through thousands of NPS surveys doesn’t sound appealing to me, and I doubt it does to you either.
Utilizing raw data uploads into a project knowledge space and having my LLM of choice run its analysis is one way to go. However, I prefer uploading this data into something like BigQuery, using LLMs to write relevant SQL queries for in-depth analysis, ensuring integrity and accuracy.
This approach not only lets me peek behind the analytical curtain, learning SQL by osmosis but also serves as a safeguard against potential inaccuracies or hallucinations often seen with direct LLM data uploads.
The separate handling of data fosters a more reliable, accurate, and actionable insight, preventing the wild goose chases that could arise from misleading automated responses.
Practically speaking, unless overwhelmed by enormous datasets, BigQuery is a free resource (setup might require a credit card, though). And fear not if SQL is new to you; with an LLM, you’re set for success with full query support in place.
Here’s a glimpse into my workflow:
Generate SQL functions using the LLM.
Debug and validate data entries.
Feed LLM with results from SQL queries.
Create visualizations either with the LLM or via further SQL queries.
Frustrations abound when attempting to secure time with subject matter experts, whose schedules often leave them stretched thin.
Why would they want to regurgitate information they’ve already discussed ad nauseam with the manufacturing team? Yet, for marketing purposes, I still need this information to clearly present new features on our platform, offering customers precise details beyond mere specifications.
How to get this coveted expertise? By crafting a customized GPT that can assume the role of interviewer, asking the right questions.
Be advised: customization may vary depending on the launch, product, or service in question. A ChatGPT Plus subscription should suffice for this task.
The guidelines should entail the following:
Role and tone: Define the interviewer’s persona.
Context: Clarify learning objectives and rationale.
Interview structure: Outline initial topics and follow-ups.
Pacing: Implement a structure of query-response dynamics.
Closing: Craft a concluding summary or call to action.
Testing it myself, I pretended to be a subject matter expert to refine this tool, always seeking to fit within their limited downtime.
The responses provided can then be further analyzed or converted into draft articles thanks to an LLM.
While potentially tricky, the strategic examination of competitors can yield profound insights regarding the competitive landscape and personal business gaps.
Here’s a few things I’ve found valuable when dissecting competitor data:
Aggregating competitors’ reviews helps identify common themes, benefits, and problem areas.
An analysis of their web copy gives clues into the type of audience they’re targeting and their unique positioning. Combine this with the Wayback Machine to track how messages have evolved over time.
Job postings can highlight strategic priorities or areas of potential experimentation.
Social media engagement data can provide insight into customer satisfaction and desire, revealing potential gaps in their customer service.
Using LLMs alongside extensive datasets allows me to remain grounded in customer realities while being swift in delivering specific, actionable insights through pair programming.
The methods explored within are just starting points. Consider other useful data sources you might already have access to:
Call transcripts from sales teams.
Query data from Google Search Console.
Insights from on-site searches.
Heatmaps tracking user interactions.
A note of caution—while analytics data is tempting, sticking to qualitative, customer-focused data rather than quantitative metrics leads to richer insights.
