What robots.txt does (and doesn't do)

robots.txt is the first file every well-behaved crawler requests when visiting your domain. It lives at the root (https://example.com/robots.txt) and tells crawlers which paths they may or may not fetch. It is a crawl directive, not an index directive: blocking a URL in robots.txt does not remove it from Google's index if other sites already link to it. To prevent indexing, use a noindex meta tag (and allow crawling so Google can see the tag).

Common robots.txt patterns

Disallow private paths: /admin, /cart, /account, internal search results (/?s=), and faceted filter URLs that generate thousands of low-value variants.
Allow CSS and JS: Google needs to render your pages. Never block /wp-includes/, /_next/static/, or your CDN paths.
Reference your sitemap: add a Sitemap: line so crawlers discover URLs faster.
Use the right user-agent: * applies to every crawler. Use specific names (Googlebot, Bingbot) only when the rule should differ.

Blocking AI crawlers in 2026

If you want to opt out of AI-training crawls or AI-search citations, you can add Disallow rules for these named user-agents - each respects robots.txt per its operator's documentation:

GPTBot - OpenAI's training crawler
OAI-SearchBot - ChatGPT search (citations)
ChatGPT-User - on-demand ChatGPT browsing
ClaudeBot and Claude-Web - Anthropic
PerplexityBot - Perplexity AI
Google-Extended - Google's AI training opt-out (separate from Googlebot)
CCBot - Common Crawl (feeds many LLMs)
FacebookBot, Bytespider, Amazonbot, Applebot-Extended

Blocking these stops ethical crawlers but does not stop scrapers; combine with server-level user-agent rules if compliance matters.

A word of caution

Disallow: / on User-agent * blocks the entire site from every crawler - the single most common SEO disaster. Always verify your robots.txt with Google Search Console's robots.txt Tester before deploying.

Frequently asked questions

Where do I upload robots.txt?

At the absolute root of your domain. example.com/robots.txt works; example.com/blog/robots.txt does not. Subdomains need their own (blog.example.com/robots.txt).

Does Crawl-delay work for Google?

No. Googlebot ignores Crawl-delay. Adjust crawl rate via Google Search Console settings. Bingbot and Yandex still honor it.

Can I block AI training but allow AI search?

Yes - for Google specifically, allow Googlebot (organic + AI Overviews) and disallow Google-Extended (Gemini training). For others, each operator documents the separation differently.

What's the difference between robots.txt and noindex?

robots.txt blocks crawling. noindex in the meta tag or HTTP header tells Google to drop the page from search. Use noindex for pages you want removed; use robots.txt to save crawl budget.