SEO & Website
Robots.txt Generator
Build a valid robots.txt file for your website, with options for WordPress and for blocking AI crawlers. Copy it or download it. Free and private.
Google ignores this setting. Some other search engines follow it.
Your robots.txt
What you enter stays on your device. The file is built in your browser. Nothing is uploaded, saved or logged.
How to use the robots.txt generator
- Choose what all crawlers may do: allow everything, block some paths, or block everything.
- Add your sitemap address, and turn on the WordPress or AI crawler options if you need them.
- Check the suggestions, then copy the file or download it.
- Upload it as
robots.txtto the root of your site, so it opens athttps://your-site.com/robots.txt.
What is a robots.txt file?
Robots.txt is a small text file at the root of your website. It tells search engine crawlers which parts of the site they may visit. Crawlers that follow the rules check it before they crawl.
It is not a way to hide pages. A blocked page can still appear in search results if other sites link to it, and anyone can open your robots.txt and read it. To keep a page out of search results, use a noindex tag or password protection instead.
Common rules
User-agent: *means the rules apply to every crawler.Disallow: /private/asks crawlers not to visit that folder.Allow: /private/page.htmlmakes an exception inside a blocked folder.Sitemap: https://example.com/sitemap.xmltells crawlers where your sitemap is.Disallow: /blocks the whole site, so use it with care.
Frequently asked questions
Is my information sent to a server?
No. The file is built in your browser, and nothing you enter is uploaded, stored or logged.
Where do I put the robots.txt file?
In the root folder of your site, so it loads at your domain followed by /robots.txt. Subfolders do not work. On WordPress, many SEO plugins, including Yoast SEO, can also edit the file for you.
Should I block AI crawlers?
It is your choice. Blocking asks AI companies’ crawlers not to use your content for training. Crawlers that respect robots.txt will follow it, but it is a request, not a lock, so some bots may ignore it.
Will blocking a page remove it from Google?
Not by itself. Robots.txt controls crawling, not indexing. To remove a page from search results, use a noindex tag or the removal tools in Google Search Console.