Free tool
robots.txt generator
Define the rules per crawler and get a valid robots.txt to drop at the root of your site. Nothing is sent to a server: everything is generated in your browser.
- robots.txt is public: anyone can read it, and it protects nothing. A path listed here is in fact advertised to everyone.
- Disallow prevents crawling, not indexing: a blocked URL that is known (through external links) can still appear in Google, without a description.
- To remove a page from Google, put a "noindex" robots meta tag in the page (or an X-Robots-Tag header) and leave it crawlable: a blocked crawler never sees the noindex.
Generated robots.txt
User-agent: * Disallow: