Category: Google

As someone deeply fascinated by how AI influences search engines, it’s intriguing to know that behind Google’s AI search facade, there is a robust system at work. This system diligently narrows down tens of thousands of documents to just a handful, relying heavily on traditional signals for visibility.

Jeff Dean, Google’s chief AI scientist, recently shared some insights on the Latent Space: The AI Engineer Podcast, where I learned how much Google’s AI still draws from its classic search engine architecture.

The architecture: filter first, reason last. In essence, for any content to be visible, it must navigate through various ranking thresholds. It starts with entering a broad candidate pool, goes through intense reranking, and only then becomes part of an AI-generated response. Essentially, AI builds on top of traditional ranking metrics.

Dean elaborated that an LLM-powered system doesn’t skim through the entire web in a single go. Instead, it begins with Google’s comprehensive index, utilizing lightweight techniques to sift through a large pool of potential documents. Dean described this process:

“You start by pinpointing a subset that seems relevant using very lightweight methods. Initially, you might have around 30,000 documents, and this number gradually refines as increasingly sophisticated algorithms and signals are applied, ultimately leading to the final 10 results or so.”

These robust ranking systems further trim this set. Consequently, it’s only after multiple filtering rounds that the most capable model steps in to analyze a significantly smaller group and generates a response. Dean continued:

“An LLM-based system isn’t vastly different. Although it processes trillions of tokens, it seeks the key 30,000-ish documents with those maybe 30 million significant tokens. From there, it derives the crucial 117 documents needed to accomplish the task.”

Dean referred to this as an “illusion” of engaging with trillions of tokens. In practice, it’s a structured pipeline: retrieve, rerank, synthesize. Dean elaborated:

“Google search isn’t about an illusion; it’s genuinely searching the internet but distilling it down to a very relevant subset.”

Matching: from keywords to meaning. Although it’s not novel, emphasizing that comprehensive topic coverage is more important than repeating exact keywords was refreshing.

Dean explicated how LLM-based representations revolutionized query-to-content matching by moving beyond word-for-word alignment. Now, Google evaluates whether pages or even paragraphs are topically relevant to a given query. He explained:

“Implementing an LLM-based text representation means we’re no longer bound by the need for specific words on a page. Instead, we delve into the topical relevance of a page or paragraph to a query.”

This paradigm shift allows Search to connect queries to answers notwithstanding different phrasings, increasingly focusing on intent and subject matter rather than mere keyword placements.

Query expansion didn’t start with AI. Dean highlighted Google’s 2001 achievement of moving its index into memory, enabling swift query expansion. He noted:

“We significantly scaled in 2001, wanting a larger index for better retrieval, accommodating growing traffic through a sharded system, evolving to fit the entire index in memory across machines. This dramatically improved query quality.”

Before this, expanding queries with additional terms was cost-intensive due to disk accesses. Once the index resided in memory, Google could enrich short queries with synonyms and variations to capture broader meanings. Dean recalled:

“Previously, term lookup was constrained by disk seek penalties. Post-memory transition, handling 50-term queries became feasible, enhancing definition and meaning extraction, far ahead of LLMs.”

This transition steered Search towards intent and semantic matching, setting the stage for today’s LLM-driven advancements, which amplify meaning-based retrieval through more refined systems and advanced computing power.

Freshness as a core advantage. Dean’s insights revealed that one of Search’s pivotal transformations involved accelerating update rates. Early on, pages refreshed monthly. Now, Google’s systems can refresh in under a minute. He observed:

“Google’s early index expansion coincided with ramping up refresh rates, now a vital parameter. Swift updates remain crucial.”

This advancement significantly enhanced news search results and overall user experience, as current data is a consumer expectation. Dean added:

“A stale index, like last month’s news, loses utility fast.”

Google’s sophisticated systems decide the frequency of page crawls, weighing potential change against the value of the latest version. Even less frequently updated important pages might be crawled often due to high update value. Dean shared:

“An intricate system determines update rates and page importance, ensuring often-updated important pages remain current.”

Why I find this crucial. The fascinating aspect is realizing that AI answers don’t bypass fundamental elements like ranking, crawl prioritization, or relevance signals. These aspects remain critical. Although LLMs reshape content synthesis and presentation, they don’t circumvent the underlying search mechanics essential for eligibility and quality.

Listen to the full interview. Discover more insights from Owning the AI Pareto Frontier — Jeff Dean.

Inspired by this post on Search Engine Land.