How to get your website cited by AI search
ChatGPT, Perplexity, Claude and Google's AI Overviews can only cite pages they can fetch and read. Here's which crawlers do what, according to each company's own documentation, and how to fix each AI search finding.
Short answer: ChatGPT search, Perplexity, Claude and Google's AI Overviews can only cite pages their crawlers are allowed to fetch and can read as plain HTML. So allow the search crawlers in robots.txt (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot), make sure your text is in the page's HTML rather than filled in by JavaScript, and write pages that answer real questions in plain sentences.
Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) is a separate choice. Each vendor's own documentation says it controls model training, not whether you appear in its search answers. An llms.txt file and FAQ markup are cheap extras, not requirements.
AI answer engines are search engines with a writer on top: a crawler indexes pages, and the assistant reads a few of them and writes an answer that links its sources. Google says AI Overviews and AI Mode use the normal Search index, with "no additional technical requirements" (Google: AI features and your website). The other assistants run their own crawlers.
robots.txt blocks AI crawlers
Fix soon Shown in the checker's AI search tab as “robots.txt blocks …” followed by the crawler names, for example “robots.txt blocks OAI-SearchBot, PerplexityBot”.
The checker reads your /robots.txt for eight crawlers. Four decide whether AI search can show and cite you: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot. Four only collect training data: GPTBot, ClaudeBot, Google-Extended and CCBot. It raises this finding only when one of the four search crawlers may not fetch your homepage, whether a group names that crawler or the User-agent: * group covers it. No robots.txt means everything is allowed.
Why it matters
The AI companies run separate crawlers for search, for user-requested fetches and for training. This table sums up each vendor's own documentation:
| robots.txt token | Company | What it does, per the vendor |
|---|---|---|
OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT's search features. Sites that block it won't be shown in ChatGPT search answers. |
ChatGPT-User | OpenAI | Visits a page when a ChatGPT user's request needs it. Not used for automatic crawling; robots.txt rules may not apply. |
GPTBot | OpenAI | Collects content that may be used to train OpenAI's models. Blocking it opts you out of training. |
Claude-SearchBot | Anthropic | Indexes content to improve Claude's search results. Blocking it may reduce your visibility in them. |
Claude-User | Anthropic | Fetches pages when a Claude user asks a question. |
ClaudeBot | Anthropic | Collects content that may be used for training. Blocking it excludes your future content from training data. |
PerplexityBot | Perplexity | Surfaces and links sites in Perplexity's search results. Not used to train AI models. |
Perplexity-User | Perplexity | Visits a page when a user asks a question. Generally ignores robots.txt because a person requested it. |
Google-Extended | Not a crawler: a token that controls whether content Google already crawls may train Gemini models and ground Gemini Apps and Vertex AI. It does not affect inclusion or ranking in Google Search, AI Overviews included. | |
Applebot-Extended | Apple | Doesn't crawl. Controls whether pages Applebot fetched may train Apple's foundation models. Pages that block it can still appear in search results. |
Bingbot | Microsoft | Bing's search crawler. Bing's AI answers draw on the Bing index. |
CCBot | Common Crawl | Builds a free, public archive of the web, which many organisations use to train AI models. |
Sources: OpenAI crawlers, Anthropic crawlers, Perplexity crawlers, Google common crawlers, About Applebot, Bing Webmaster blog and Common Crawl: CCBot.
The serious case is Disallow: / under User-agent: *, or a block on PerplexityBot or Bingbot: those take you out of search-style answers. Blocking GPTBot, ClaudeBot, Google-Extended or CCBot is about training, a legitimate business choice, so the checker shows those in yellow as “blocked (AI training only)” rather than raising the finding.
How to fix it
For most businesses that want to be found, the simplest choice is to allow everything. To opt out of training but stay citable, use separate groups:
# Search and answer engines: allowed to read and cite the site
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
# Optional: opt out of AI model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /
Sitemap: https://yourbusiness.com/sitemap.xmlThe checker will still list deliberately blocked training crawlers, in yellow. OpenAI and Perplexity both say changes take up to about 24 hours. Our robots.txt and sitemap guide covers the file itself.
WordPress
WordPress serves a virtual robots.txt unless a real file exists in the root folder. Most SEO plugins include an editor; otherwise upload the file with your host's file manager or SFTP. Security plugins may add their own bot blocks.
Cloudflare
Cloudflare can block AI crawlers and prepend managed lines to your robots.txt. Look under Security Settings → Configure AI bot policies. From September 15, 2026, Cloudflare's defaults for new domains block training and agent bots on pages that show ads (Cloudflare: block AI bots). A firewall block doesn't appear in robots.txt, so the checker can't see it.
Wix, Squarespace and Shopify
Wix has a robots.txt editor in its SEO settings, and Shopify lets you customise the file through a robots.txt.liquid theme template. Squarespace doesn't allow direct edits, but look in its crawler settings for an option that blocks AI crawlers, and switch it off if you want to be cited.
How to check it's fixed
Open https://yourbusiness.com/robots.txt in a private window and read what's actually served, including any Cloudflare lines. Then run the website checker again; the AI search tab lists each crawler as allowed, blocked or not mentioned.
No llms.txt file
Minor Shown in the checker's AI search tab as “No llms.txt”.
The checker requests /llms.txt and flags it if the request doesn't return 200, if the file is empty, or if the server returns an HTML page instead (common when a missing file falls through to the homepage).
Why it matters (honestly)
llms.txt is a proposed convention from 2024, not a standard. It's a short Markdown file at your site's root that summarises who you are and links to your key pages, so far used mostly for software documentation. None of the crawler pages from OpenAI, Anthropic or Perplexity say their crawlers read it, and Google says you don't need "AI text files" to appear in AI features. Treat it as a fifteen-minute extra that may help agents and can't hurt.
How to fix it
Create llms.txt in the same folder as robots.txt: an H1 with your name, a summary in a blockquote, optional detail, then H2 sections of links:
# Smith Plumbing
> Licensed plumbers in Yonkers, NY. Repairs, water heaters and drain
> cleaning for homes and small businesses in Westchester County.
Open Monday to Saturday, 7am to 6pm. Call (914) 555-0100.
## Main pages
- [Services](https://smithplumbing.example/services/): what we fix and install
- [Service area](https://smithplumbing.example/area/): towns we cover
- [Contact](https://smithplumbing.example/contact/): phone, email and addressOn WordPress or cPanel hosting, upload it with the file manager or SFTP. Wix, Squarespace and Shopify may not allow arbitrary files at the root, so check their help first. Keep the facts identical to your site and Google Business Profile.
How to check it's fixed
Open https://yourbusiness.com/llms.txt. You should see your plain text, not your homepage or a 404 page.
The page's text is built by JavaScript
Fix soon Shown in the checker's AI search tab as “Only about N words in the HTML; the page is built by JavaScript”.
The checker counts the visible words in the HTML your server sends, before any script runs. It flags pages with under 60 words, at least one script and an empty app container (a div or main with an id such as root, app or __next), or pages with under 25 words and two or more scripts.
Why it matters
Google renders JavaScript, but in a separate queue that can delay indexing, and its own documentation adds that "not all bots can run JavaScript" (Google: JavaScript SEO basics). None of the OpenAI, Anthropic or Perplexity crawler pages say their bots run scripts, so the safe assumption is that they read only the HTML. To them, an app-shell page is nearly empty.
How to fix it
Put the words in the HTML. On a React or Vue site, render on the server or pre-build static pages (Next.js, Nuxt and Astro all do this); Google calls dynamic rendering a workaround, not a fix. WordPress, Wix, Squarespace and Shopify normally send their text in the HTML already. If yours don't, look for a page builder or app that loads content by script.
How to check it's fixed
Run curl -s https://yourbusiness.com, or use your browser's “View page source” (not the inspector), and look for your headline and paragraphs in the raw HTML.
No FAQ, HowTo or Article structured data
Minor Shown in the checker's AI search tab as “No FAQ, HowTo or Article structured data”.
The checker looks through the page's JSON-LD for one of FAQPage, HowTo, Article, NewsArticle, BlogPosting, TechArticle, QAPage or Question, and flags the page when none is present.
Why it matters
These types label questions, answers and articles for machines. Be realistic about what that buys: Google says no special schema is needed for AI features, and it stopped showing FAQ rich results in Search on May 7, 2026. What gets quoted is your visible text; the markup only labels it, and it must match what's on the page.
How to fix it
Add a short FAQ section with the questions customers really ask, then describe it with FAQPage JSON-LD that repeats exactly the same questions and answers. Use Article or BlogPosting on blog posts and guides. On WordPress, SEO plugins and FAQ blocks generate it. Never mark up questions that aren't visible. Our structured data guide has full examples.
How to check it's fixed
Paste the page address into the Schema.org validator. It should list the FAQPage or Article item with no errors. For Article markup, Google's Rich Results Test also works.
No question-style headings
Minor Shown in the checker's AI search tab as “No question-style headings”.
This appears only when the page has none of the structured data above and no H2, H3 or H4 heading that ends in a question mark.
Why it matters
People ask assistants full questions, such as “who fixes water heaters near Yonkers on a Saturday?” A question heading followed by a direct answer is a passage an assistant can lift and cite.
How to fix it
Add a “Common questions” section with three to six real questions as H2 or H3 headings (price ranges, service area, hours, how long a job takes), taken from your calls and emails. Answer each in the first sentence, then add the FAQPage markup above.
How to check it's fixed
Run the checker again. This finding and the structured-data one should both clear.
Sources
- OpenAI: overview of OpenAI crawlers
- Anthropic: does Anthropic crawl data from the web, and how can site owners block the crawler?
- Perplexity: Perplexity crawlers
- Google: common crawlers, including Google-Extended
- Google Search Central: AI features and your website
- Apple: about Applebot
- Common Crawl: CCBot
- llmstxt.org: the /llms.txt proposal
Common questions
Should I block GPTBot?
Only if you don't want your content used to train OpenAI's models. OpenAI says GPTBot is for training and OAI-SearchBot is for ChatGPT search, and each setting is independent, so you can block GPTBot and still appear in ChatGPT search answers by allowing OAI-SearchBot.
Does blocking Google-Extended remove my site from AI Overviews?
No. Google says Google-Extended doesn't affect inclusion or ranking in Google Search. AI Overviews and AI Mode use the normal Search index, controlled by Googlebot and snippet controls such as nosnippet.
Do I need an llms.txt file to show up in ChatGPT or Google?
No. llms.txt is a proposed convention, and Google says you don't need AI text files to appear in its AI features. It's a cheap extra that some AI agents and tools read, not a requirement.
How do I know if ChatGPT or Perplexity can read my website?
Check that robots.txt allows OAI-SearchBot and PerplexityBot, that your firewall or CDN isn't blocking them, and that your text appears in the raw HTML (curl or View page source). The website checker tests robots.txt and the HTML in one pass.
Is there a way to see when AI answers cite my site?
Google counts AI Overviews and AI Mode traffic in Search Console's Performance report under the Web search type. Bing Webmaster Tools added an AI Performance report (a public preview since February 2026) that shows citations in Microsoft Copilot and Bing's AI summaries.
Does structured data help AI assistants cite me?
It labels your content clearly, but Google says no special schema is needed for its AI features. Write clear question-and-answer text on the page first, then add FAQPage or Article markup that matches it exactly.
See what AI crawlers see. The website checker reads your robots.txt for each AI crawler, looks for llms.txt, counts the text in your HTML and checks your structured data, alongside security, speed, SEO and email. If you'd rather have it handled, we can set it up for you.