Groups and rules
A robots.txt file sits at the root of a site and tells crawlers which paths to stay out of. Each group names one or more user agents and lists Disallow and Allow rules; a crawler follows the group that names it most closely and ignores the rest, falling back to *. An empty Disallow: allows everything, and Disallow: / asks to stay out of the whole site. Paths start with /, and one that does not is marked.
Blocking AI crawlers
Block AI crawlers adds a group for 19 crawlers that collect pages to train or feed AI models, including GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended and PerplexityBot, without changing what search engines may index. Press it again after editing and the group is replaced rather than added twice.
Sitemap and saving
Add the full address of your sitemap and a Sitemap: line goes at the end. Download saves the file as robots.txt; it belongs at https://your-site/robots.txt, and each subdomain needs its own.
Questions
Does robots.txt keep a page private?
No. It is a request that well-behaved crawlers follow, not a lock, and the file itself is public, so listing a secret path in it advertises the path. Use a password or a noindex tag for pages that must stay out of search.
Will blocking AI crawlers hurt my search ranking?
Not in Google or Bing. Google-Extended only controls whether Google uses your pages for its AI models; Googlebot, which ranks pages, is a separate agent and is not blocked. New AI crawlers appear often, so check the list now and then.