The crawlers that feed each engine
In short
Each answer engine reads the web through its own crawler, and a site that blocks that crawler in robots.txt should expect to appear less, or not at all, in that engine's answers. Where an engine also collects pages for training, it uses a separate crawler, so a site can refuse training and still be found.
OpenAI says OAI-SearchBot is the crawler that surfaces websites in ChatGPT's search features, and that sites which opt out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. Its GPTBot collects pages that may be used to train models, and ChatGPT-User visits a page when a person asks ChatGPT to, so robots.txt rules may not apply to it. OpenAI adds that a robots.txt change can take about 24 hours to reach its systems.
Perplexity runs PerplexityBot to surface and link websites in its search results and says it is not used to crawl content for AI foundation models; its Perplexity-User fetcher follows a person's request and generally ignores robots.txt. Anthropic names ClaudeBot for collecting content for its models, Claude-User for pages a person asks Claude to read, and Claude-SearchBot for search, and says that blocking Claude-SearchBot may reduce a site's visibility in Claude's search results.
Google works differently. AI Overviews and AI Mode draw on the ordinary Search index that Googlebot builds, so there is no separate AI crawler to let in. Google-Extended is a control in robots.txt for training and grounding in Gemini models and apps, and Google says it does not affect a site's inclusion or ranking in Google Search.
| Feeds its search answers | Collects for training | |
|---|---|---|
| ChatGPT (OpenAI) | OAI-SearchBot | GPTBot |
| Perplexity | PerplexityBot | None named: Perplexity says PerplexityBot is not used to train models |
| Claude (Anthropic) | Claude-SearchBot | ClaudeBot |
| Google AI Overviews and AI Mode | Googlebot, the ordinary Search crawler | Google-Extended, a robots.txt control rather than a crawler |
Swipe sideways to see every column
How an engine chooses among the pages it can read
In short
The engine turns a question into searches of its own, often several, and writes from the passages that answer them best. No engine publishes the rules it uses, and the same question can bring different sources on a different day.
Google's guide to its generative features says they are rooted in its core ranking and quality systems: a model retrieves pages from the Search index, reviews what they say and shows links to the pages that support its response. The same guide says it adds no special requirements of its own, so the pages it can cite are the pages Search already indexes.
Research points the same way. A study by researchers at Princeton and elsewhere, accepted to the KDD 2024 conference, rewrote pages in different ways and measured how much of each one a generative engine used. Adding citations to sources, quotations and statistics gave relative gains of 30 to 40 percent on one of its main measures, while keyword stuffing, common in classic SEO, gave little or no gain.
Those tests ran on a research setup built on a GPT-3.5 model and on Perplexity, not on every engine people use today, so treat them as a direction rather than a rule. A passage that states a specific fact, says where the fact comes from and answers the question without leaning on the paragraphs around it is easier for any engine to use.
What you control, and what you do not
In short
You control whether the crawlers may come in and how clearly your pages answer. You do not control which sources an engine picks for a given question, and no engine sells a place among the sources its answers cite.
The parts in your hands are practical. Keep your pages public and crawlable, with the words in the page itself rather than inside images, list them in a sitemap, and keep the facts about your business the same on your site and your listings.
Google's guide says there is no need to cut content into tiny chunks or to write in a special way for AI, and that Google Search ignores AI text files such as llms.txt. It warns that seeking inauthentic mentions across the web is not as helpful as it might seem, and that no third-party tool has access to its internal ranking or AI systems, so be wary of any tool or service that promises you a place in the answers.
Search Console shows part of the picture for Google. Since August 31, 2026 its generative AI performance report has been open to sites worldwide: it counts how often links to a site appeared in AI Overviews and AI Mode, page by page. A setting beside it lets an owner take a site out of those features, and every site stays in them unless its owner opts out.
FAQ markup and llms.txt: kept, not relied on
In short
Both cost little and neither is a lever. Google stopped showing FAQ rich results from May 7, 2026, and says Google Search ignores llms.txt files, so a site keeps them for other readers rather than for a place in Google's results.
FAQ markup labels a page's questions and answers as structured data. Google once showed it as expandable questions under a result, then only for a few government and health sites, and its documentation changelog says the feature no longer appears in Google Search starting May 7, 2026. The markup still describes the page truthfully, as long as it matches the questions a reader can see.
llms.txt is a proposed file that summarizes a site for AI tools. Ahrefs checked 137,000 domains and found that of the roughly 38,000 with a valid file, 97 percent saw no requests for it at all in May 2026. Google says keeping one will neither help nor harm a site in Google Search, so it is a cheap extra rather than a strategy.
What this means for a small business
In short
Let the search crawlers in, then spend your effort on pages that answer your customers' questions plainly. Measure it by asking the engines those questions yourself and noting which sources they name.
Open your robots.txt, or ask whoever runs your site, and check that OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot are allowed. If you would rather OpenAI, Anthropic and Google not train their models on your pages, you can block GPTBot, ClaudeBot and Google-Extended and still appear in ChatGPT search, Claude's search and Google Search, though Google-Extended also keeps your pages out of grounding in Gemini's own apps.
Then look at the questions a customer asks before calling you, such as what you charge, how long a job takes or whether you serve their part of town. A page that answers one of them in its first lines, in words a stranger understands, gives an engine something it can quote.
For a business that serves one city, such as a shop or a practice in Los Angeles, Google adds one thing: details in a Business Profile can feed its AI answers about local businesses. Keep the profile and the site in agreement, down to the hours and the services you list.