A robots.txt file tells search engine crawlers which parts of your site they may visit. Pick a preset, add your own rules and sitemap, then copy the result or download a ready-to-upload robots.txt file. It runs in your browser and nothing you enter is stored.
Robots.txt Generator
Create a robots.txt file with WordPress and WooCommerce presets, AI crawler blocking and your sitemap URL.
Input Section
Result Section
Insights / Tips
- Robots.txt controls crawling, not indexing. Use a noindex tag to keep a page out of search results.
- Never block CSS or JavaScript files that your pages need to render.
- Only well-behaved crawlers follow robots.txt. It is not a security tool.
Internal Links
What this tool does
- Presets: WordPress, WooCommerce, allow everything, or block everything.
- WooCommerce rules: keeps crawlers out of cart, checkout, account and filter URLs.
- Custom rules: add your own Disallow and Allow paths, one per line.
- AI crawler blocking: one switch adds rules for common AI training crawlers such as GPTBot, ClaudeBot, CCBot and PerplexityBot.
- Sitemap line: adds your sitemap URL so crawlers find it quickly.
- Internal search blocking: stops crawlers from wasting time on WordPress search result pages.
How to use the robots.txt generator
- Choose a preset that matches your site.
- Enter your sitemap URL, for example
https://example.com/sitemap_index.xml. - Add any extra paths you want to block or allow. Each path must start with
/or*. - Click Generate robots.txt.
- Download the file and upload it to the root folder of your site, so it opens at
yourdomain.com/robots.txt. - Test it in Google Search Console’s robots.txt report.
Important things to know
- Robots.txt controls crawling, not indexing. A blocked page can still appear in search results if other sites link to it. Use a noindex tag to keep a page out of results.
- Do not block CSS or JavaScript that your pages need, because search engines use them to render the page.
- It is not a security tool. Only well-behaved crawlers follow it, and anyone can read the file.
- Blocking AI crawlers only affects crawlers that respect robots.txt. Blocking Google-Extended does not affect your Google Search rankings.
- Crawl-delay is ignored by Google. Bing and some other crawlers respect it.
- WordPress creates a virtual robots.txt if no file exists. A real file in your root folder replaces it.
Frequently asked questions
What is a robots.txt file?
It is a text file in your site’s root folder that gives crawlers rules about which URLs they can request.
Where do I upload robots.txt?
In the root folder of your site, usually public_html, next to wp-config.php. It must be reachable at https://yourdomain.com/robots.txt.
What should WordPress robots.txt contain?
A common setup blocks /wp-admin/ but allows /wp-admin/admin-ajax.php, and lists your sitemap. The WordPress preset does exactly this.
Should I block /wp-content/ or /wp-includes/?
No. Search engines need the CSS, JavaScript and images stored there to render your pages.
Will blocking AI bots stop my content being used?
It asks compliant crawlers not to collect it. It cannot stop crawlers that ignore robots.txt.
Can I have more than one Sitemap line?
Yes. You can add extra Sitemap lines by hand after downloading the file.