robots.txt Generator
Generate a robots.txt file for your website. Control which pages search engines can crawl.
Output
Quick Presets
About
The robots.txt file tells search engine crawlers which pages they can or cannot access on your site. Placed at the root of your domain (e.g., example.com/robots.txt), it is one of the first files crawlers check. A well-configured robots.txt helps manage your crawl budget and prevent indexing of private or duplicate content.
FAQ
Where should I place robots.txt?
Place it at the root of your domain: https://yourdomain.com/robots.txt
Can I block AI crawlers?
Yes. Use the Block AI Bots preset to disallow GPTBot, ChatGPT-User, Anthropic-ai, and CCBot.