How to Rank Your Website in the AI Era: The Complete 2026 SEO & AI Search Framework

How search actually works in 2026

Google's search architecture is a pipeline of crawling, indexing, scoring, and re-ranking systems, not one static calculation. The names below come from the Content Warehouse API documentation that leaked in 2024, as analyzed by Mike King at iPullRank.

Web discovery begins with the Trawler crawler, which fetches pages and routes them to Alexandria, the core indexing system, which builds the CompositeDoc, the record for an indexed URL. Nested inside it, the PerDocData model holds page-level quality and classification attributes.

The documentation also describes index tiers: the most important documents are kept on the fastest storage and the rest on slower tiers. A link attribute called sourceType records which tier the linking page sits in, and King reads that as links from higher-tier pages being worth more.

For the full 28-year history of how this pipeline was built, see our complete timeline of Google's search algorithm.

At query time, Mustang is the primary scoring, ranking, and serving system. A DOJ trial exhibit, the notes of a February 2025 call with Google engineer HJ Kim, describes the main scores. Topicality (T*) combines three fundamental signals: anchors (links), body (terms in the document), and clicks (how long a searcher stayed before returning to the results). Quality (Q*) is described as largely static and tied to the site more than the query.

PageRank is described as a single signal relating to distance from a known good source, used as an input to the quality score. The leaked documentation lists siteAuthority as an attribute "applied in Qstar", next to demotion fields such as pandaDemotion and navDemotion.

Many analysts read that as a site-level quality gate. The documents show the attribute exists, not how it is computed or how much it counts.

Initial scores are then adjusted by Twiddlers, re-ranking functions that run after the primary scoring. NavBoost is the click-based one. Trial testimony from Google's Pandu Nayak, as reported by iPullRank, put its window at about 13 months of click data, down from 18. FreshnessTwiddler re-ranks documents based on freshness, and QualityBoost is listed among the boost functions. A separate attribute, contentEffort, is described in the documentation as an LLM-based effort estimate for article pages.

How AI engines retrieve and cite you

AI search platforms use retrieval-augmented generation (RAG): they retrieve web documents, extract the relevant passages, and generate an answer with citations. What you can control starts with the crawler. OpenAI's documentation says OAI-SearchBot is used to surface websites in ChatGPT's search features, and that sites opted out of it will not be shown in ChatGPT search answers. GPTBot is a different crawler that collects content for model training, and each setting is independent of the others.

Perplexity documents the same split: PerplexityBot surfaces and links sites in its search results, while Perplexity-User fetches a page when a user's question calls for it. Google's guidance is that a page must be indexed and eligible to be shown with a snippet to appear in its generative AI features.

What happens after retrieval is not documented by the vendors, so treat the following as one study's finding. In July 2026 RESONEO analyzed 1,249 ChatGPT answers and found a result_source field with four values: labrador, bright, oxylabs, and serp.

RESONEO reads labrador as OpenAI's in-house index, bright as scraped Google results, and oxylabs as a news pipeline. It also reports that the field disappeared from responses on 21 July 2026. Those labels are inferences from network data, not anything OpenAI has confirmed, and they can change without notice.

Our complete AEO checklist turns this section into steps you can work through.

E-E-A-T and what the leaked docs show

Experience, Expertise, Authoritativeness, and Trust (E-E-A-T) is the framework Google gives its human quality raters. Their ratings evaluate search results and do not rank pages directly. The guidelines' change log records a January 2025 update that revised the Lowest and Low quality sections to align with Google's spam policies.

The current edition defines generative AI, tells raters to give the Lowest rating when they strongly suspect scaled content abuse, however the pages were made, and treats filler as a poor user experience. Trust is the central pillar, and experience, expertise, and authority support it.

The leaked documentation suggests related machine signals exist. PerDocData lists authorObfuscatedGaiaStr, an obfuscated identifier for a document's author, and iPullRank notes that Google stores authors as text and has an isAuthor attribute for entities.

PerDocData also lists ymylHealthScore and ymylNewsScore, described as scores from Your-Money-or-Your-Life (YMYL) health and news classifiers. The documentation does not say how any of these feed ranking. The practical reading is modest: Google can tell who wrote a page and whether its topic is sensitive, so make both easy to verify.

Information Gain: the new quality bar

Generative tools have made paraphrased text trivial to produce at scale, which is why Information Gain matters. The term comes from a Google patent, "Contextual estimation of link information gain", granted in June 2022. It describes scoring how much additional information a document offers beyond the documents a searcher has already been shown, and using that score to rank or demote results.

A patent shows what Google has considered, not what is running in production, and Google has not confirmed using it.

The idea is simple. If a document only rewrites the pages that already rank, the overlap is nearly total and it adds almost nothing. If it brings its own statistics, benchmarks, or expert quotes, the overlap shrinks and the page has something a searcher, or an AI answer, can't get elsewhere. To audit content against this standard, we use a five-part rubric of our own:

Entity SEO and semantic embeddings

Search engines work with entities (people, companies, products, places) and the relationships between them, not only with keywords. In the leaked documentation, Google's WebRef system annotates the entities found in a page's text.

Analysts point to attributes such as confidenceScore (how certain the system is that a phrase is a known entity), topicalityScore (how relevant that entity is to the page), and segmentMentions (the entity's mentions across the document).

Entities connect into subject-predicate-object statements such as [ChatGPT] → is_developed_by → [OpenAI], which is how a knowledge graph stores facts.

The documentation also references page-level and site-level embeddings, which are numeric representations of what a page or a whole site is about. Two attributes build on them. iPullRank describes siteFocusScore as how much a site sticks to a single topic, and siteRadius as how far a page goes outside that core topic.

Analysts read this as a reason to keep a site on-topic. The documents do not show a demotion for drifting, so treat that part as a reasonable inference.

Brand authority and the AI recommendation engines

When someone asks an AI assistant to recommend a product or provider, the answer names a handful of brands. The SEO agency First Page Sage ran 11,128 commercial queries across ChatGPT, Gemini, Perplexity, and Claude, with its latest data gathered between March and July 2026, and estimated what drives those recommendations. Its answer is not backlink counts.

The largest factors were mentions in authoritative lists, awards and accreditations, and online reviews. These are one agency's modeled estimates, not figures published by the AI companies.

For ChatGPT, First Page Sage puts authoritative list mentions at 41% of recommendation weight. For Gemini's general recommendations it puts list mentions at 49% and a factor it calls Google website authority at 23%.

For local queries it found that a high star rating on Google Business Profile correlates most strongly with being recommended. It also reports that Gemini does not recommend companies with low online reviews, which it defines as under 3.5 stars. That is the study's observation, not a rule Google has published.

The practical lesson is that reputation compounds. Useful work and honest digital PR get a brand named in the places people and AI systems consult, and more people then search for it by name. Those mentions have to be earned: Google's spam policies prohibit buying links or manufacturing them at scale, and a list you paid to appear on is not the signal these systems are looking for.

NavBoost, Chrome data, and Core Web Vitals

Click data is one of the better-documented parts of the system. In the DOJ exhibit, Google engineer HJ Kim lists clicks as one of three fundamental signals. The leaked documentation lists the click attributes NavBoost works with, including goodClicks, badClicks, and lastLongestClicks.

Analysts read these as clicks that satisfied the searcher, quick returns to the results page (pogo-sticking), and the last, longest click of a session. Chrome is less clear. The documentation lists chromeInTotal, described as site-level Chrome views, and the DOJ exhibit refers to a popularity signal that uses Chrome data. Neither source says how that data is weighted.

The safe conclusion is that what visitors do after the click matters, so a page should load quickly and answer the query it ranks for. Core Web Vitals are Google's public measure of that experience. A page passes when it meets all three thresholds:

How backlinks are actually evaluated in 2026

Links still count, and the evidence points to where they come from mattering more than how many there are. In the DOJ exhibit, HJ Kim describes PageRank as a single signal relating to distance from a known good source, used as an input to the quality score.

The leaked documentation has a matching attribute, homepagePagerankNs, which Mike King reads as the "nearest seed" version of PageRank. Under that reading, Google starts from a set of trusted seed sites and measures how many link hops separate a page from them.

Pages a few hops from a trusted source inherit more trust than pages far out in the graph. Google has not published which sites are seeds.

The leaked documentation also shows Google recording the quality of the page a link comes from. The sourceType attribute on an anchor marks whether the linking page sits in the high, medium, or low quality part of the index.

A separate field, anchorMismatchDemotion, points to a demotion when a link doesn't match the page it points to. That fits the reason a direct, reciprocal link exchange still carries risk today: it isn't the existence of a link that gets flagged, it's the pattern the link leaves behind.

For the kinds of pages that attract links on their own, see how to get backlinks without outreach.

Technical SEO for AI crawlers

If crawlers can't reach and parse a page, it stays invisible to search engines and AI answer engines alike. A common mistake is blocking AI search bots by accident, through a broad robots.txt rule or a CDN or firewall setting that blocks AI bots by default. Each AI company runs separate bots for search and for model training, and robots.txt controls them independently.

Per OpenAI's documentation, OAI-SearchBot decides whether you can appear in ChatGPT search, while GPTBot only governs training. Anthropic splits Claude-SearchBot (search) from ClaudeBot (training) the same way. To be citable, allow the search bots:

# Google Search, including AI Overviews and AI Mode
User-agent: Googlebot
Allow: /

# ChatGPT search results (OpenAI's search bot)
User-agent: OAI-SearchBot
Allow: /

# Perplexity search results
User-agent: PerplexityBot
Allow: /

# Claude search results (Anthropic's search bot)
User-agent: Claude-SearchBot
Allow: /

# Training crawlers are a separate decision.
# GPTBot (OpenAI) and ClaudeBot (Anthropic) collect content for model
# training. Allowing or blocking them does not change whether the
# search bots above can surface your pages.

To check your own site, open yourdomain.com/robots.txt and look for a Disallow rule under any of those four names, or under User-agent: *. OpenAI notes that a robots.txt change can take about 24 hours to take effect.

Rendering is the second thing to check. Google can render JavaScript, but don't assume AI crawlers will. If pricing, specifications, or key facts load only after client-side rendering, a bot that reads the raw HTML never sees them.

Server-side rendering (SSR) or static site generation (SSG) puts that content in the initial HTML response. View the page source, not the inspector, and confirm the text is there. Structured data then describes that content in a machine-readable way.

Google says it isn't required for its generative AI features and that no special markup is needed, but it still supports rich results and makes clear who you are. These are the types worth having in JSON-LD:

Google's stance on AI-generated content

Google's official guidelines state that appropriate use of automation or AI generation doesn't violate its policies, provided the content is created primarily to help people rather than manipulate rankings. If a site uses AI to mass-produce low-effort pages mainly to capture search traffic, that is scaled content abuse under Google's spam policies, and the site can rank lower or be removed from results.

Be wary of claims that a specific update added an "AI detector". Google's Search Status Dashboard records the March 2026 core update as rolling out from March 27 to April 8, 2026, and says nothing about AI content or any named model.

What Google has put in writing is narrower. Its guide to generative AI features says content made with AI tools must still meet the Search Essentials, and that creating many pages mainly to manipulate AI responses violates the scaled content abuse policy.

The quality rater guidelines add that such pages deserve the Lowest rating no matter how they were created.

To scale content responsibly, use a hybrid model: let generative tools cluster topics and draft outlines, then require a human subject-matter expert to review and edit every draft, integrating proprietary data, case studies, first-person insight, and verifiable quotes. Avoid publishing automated translations or mass keyword variants without manual review, so every page serves a clear, real user intent.

Where SEO is headed next, the 3 to 5 year horizon

As AI models develop agentic capabilities, queries are shifting from informational lookups to task execution. Agentic search tools can autonomously browse the web, parse multiple sources, and execute multi-step processes like booking flights, purchasing products, or compiling market intelligence on a user's behalf. SEO strategies need to optimize a brand's technical structure for these automated agents, not just for human screen layouts.

Search behavior is also diversifying away from a single web-text input. Searchers increasingly use image search to search by photo, and use video platforms directly. AI models work across text, images, and video, so a useful video that answers the question is another way to be found. Published figures on how often AI engines cite video sites vary widely between studies, so we don't quote one here.

Finally, expect highly personalized context graphs to replace identical results for every user. With permission, models like Gemini can draw on email, cloud storage, calendar events, and browsing history to understand intent in real time, meaning search results and citations get personalized around a specific workflow, prior searches, and immediate location context.

Strategic synthesis: the priority framework

Ranking factors don't carry equal weight, and nobody outside Google knows the real weights. This framework is our own ordering. Each row says what the evidence is, so you can judge how much to trust it:

As an organization's search strategy matures, its focus, technical actions, and content standard shift through four stages:

Every layer below reinforces the one above it. Skipping the foundation to chase AI readability doesn't work, because a technically broken site never earns the chance to be judged on its content.

What Google says versus what the leak lists

Google's public statements have often been simpler than the systems behind them. The left column paraphrases positions Google representatives have taken over the years, and the middle column shows what the leaked documentation and DOJ exhibits list. Read it with the same caveat HJ Kim gave the DOJ: the leak named components of the ranking system, but not the curves and thresholds that decide how they are used.

What to do, by business size and site type

The right priority also depends on your business model. Here's how the primary quality metric, biggest technical risk, and target off-page link differ by site type:

Small and local businesses

Mid-market and SaaS businesses

Enterprise and global brands

Frequently asked questions

Glossary of modern search & AI terms

Every acronym and internal attribute referenced above, in one place.

Alexandria

The core Google database responsible for indexing web documents and building the CompositeDoc.

AEO (Answer Engine Optimization)

Formatting content specifically to be extracted and presented as direct answers by AI platforms and voice assistants.

ClutterScore

A leaked site-level attribute, described in the documentation as penalizing sites with a large number of distracting or annoying resources.

CompositeDoc

Google's master indexed record for a single URL, containing its parsed content, canonical links, and PerDocData metadata.

GEO (Generative Engine Optimization)

Optimizing web assets so they're easily synthesized, parsed, and cited within AI-generated search overviews.

GoodClicks / BadClicks

Click attributes listed in the leaked documentation. Analysts read them as satisfied clicks versus quick returns to the results page.

Information Gain

A score described in a Google patent for how much new information a document adds beyond documents the searcher has already seen.

INP (Interaction to Next Paint)

A Core Web Vital evaluating visual interaction speed, requiring responsiveness under 200 milliseconds to pass.

Labrador

A result_source label seen in ChatGPT's network responses until July 2026. RESONEO's analysis reads it as OpenAI's in-house search index. OpenAI has not documented it.

Mustang

Google's primary scoring, query-matching, and initial candidate retrieval engine.

NavBoost

Google's click-based re-ranking system. DOJ trial testimony put its window at roughly 13 months of click data.

OriginalContentScore

A leaked PerDocData attribute scoring originality on a 0 to 127 scale. iPullRank notes it appears on pages with little content.

PerDocData

The dynamic quality-scoring table nested within a CompositeDoc, storing page-level quality, freshness, and classification flags.

Query Fan-Out

An AI retrieval technique where a single complex request is broken into multiple targeted queries executed in parallel.

Q* (Q-Star / siteAuthority)

Q* is Google's quality score, described in a DOJ exhibit as largely static and tied to the site. The leaked siteAuthority attribute is described as applied in Q*.

SiteFocusScore

A leaked attribute for how closely a site sticks to a single topic.

SiteRadius

A leaked attribute for how far a page strays from its site's core topic.

T* (Topicality)

Google's relevance score for a document against a query, built from anchors, body text, and clicks, per a DOJ exhibit.

Trawler

Google's web crawler, responsible for discovering and fetching web pages.

Twiddlers

Real-time ranking-adjustment filters that run after primary scores are calculated, applying modifiers for freshness, localization, or engagement.

WebRef

Google's machine-learning model that extracts entities from web text and links them to the Knowledge Graph.

More of the pipeline can be mapped now. The weights are still hidden, but nothing in the documents rewards manufactured signals over a useful site.