Robots.txt Generator
Create a custom robots.txt file for your website with crawl rules for Google, Bing, GPTBot and other bots.
Fill in details above to generate your robots.txt
What is a robots.txt file?
The robots.txt file is a plain text file placed at the root of your website (e.g., https://example.com/robots.txt) that tells search engine crawlers and other bots which pages or sections of your site they are allowed to crawl. It is part of the Robots Exclusion Protocol (REP) and is checked by Google, Bing, and most other crawlers before they index your site.
How to use this robots.txt generator
- Enter your sitemap URL (highly recommended for SEO).
- Add crawl rules for different user agents (bots).
- Optionally block AI training crawlers like GPTBot or ClaudeBot.
- Copy or download the generated robots.txt file and upload it to the root of your website.
Does robots.txt prevent pages from being indexed?
No. Blocking a URL in robots.txt prevents crawlers from visiting it, but the URL may still appear in search results if other pages link to it. To prevent indexing, use a noindex meta tag or HTTP header instead.
Should I block AI crawlers?
This is a personal choice. Blocking AI crawlers (GPTBot, ClaudeBot, etc.) prevents your content from being used to train AI models. It does not affect Google or Bing search indexing. Many publishers now opt to block AI crawlers to protect their content.
Where do I put my robots.txt file?
It must be placed at the root of your domain: https://yourdomain.com/robots.txt. It will NOT work from a subdirectory.
What is Crawl-delay?
The Crawl-delay directive asks a crawler to wait a specified number of seconds between requests to reduce server load. Note: Googlebot ignores Crawl-delay and has its own crawl rate settings in Google Search Console.