Free tool

robots.txt generator

Define the rules per crawler and get a valid robots.txt to drop at the root of your site. Nothing is sent to a server: everything is generated in your browser.

  • robots.txt is public: anyone can read it, and it protects nothing. A path listed here is in fact advertised to everyone.
  • Disallow prevents crawling, not indexing: a blocked URL that is known (through external links) can still appear in Google, without a description.
  • To remove a page from Google, put a "noindex" robots meta tag in the page (or an X-Robots-Tag header) and leave it crawlable: a blocked crawler never sees the noindex.

Generated robots.txt

User-agent: *
Disallow:

This site only uses cookies that are necessary for it to work, including one to remember that you have read this message. We also measure our traffic with a tool we host ourselves, without cookies and without storing your IP address. No advertising, no third-party trackers. Learn more about privacy