Robots.txt generator
Name the crawler, list paths to allow or block, and add the sitemap address. Copy the file or download it.
- A file you can save
- Free, no sign-up
Free to use, no account. More seo tools
How to use it
- Set the user-agent. * means every crawler.
- Add one path per line. A leading slash is added if you leave it off.
- Copy the result into the site root as robots.txt.
What the lines mean
Allow and Disallow name paths, not whole sentences. An empty disallow list with no allow lines writes Disallow with nothing after it, which means the agent may fetch everything. Crawl-delay is written if you fill it. Google’s crawler ignores crawl-delay. Other crawlers may still read it.
The sitemap line is a full URL. The page does not check that the file exists.
Private by design
Privacy
The paths you type stay in this tab and are not uploaded.
Questions
Does this block a page from the index?
No. robots.txt asks a crawler not to fetch a path. A noindex tag is a different instruction. Use the meta tag generator for that.
Can I write a rule for one bot?
Yes. Change * to the crawler name, such as Googlebot. A second bot needs a second file block, which you can paste underneath.
Where does the file go?
In the host’s root, so the address is /robots.txt.