Free robots.txt generator for WordPress.
Build a valid robots.txt file using checkboxes and templates. Choose your CMS, select which bots to block, add custom rules, and copy the result. No account required.
1. Start from a template
2. Add your sitemap URL
Rank Bix generates your sitemap at /sitemap.xml. Add this line so all crawlers find it automatically.
3. Select what to block
4. Block problem bots (optional)
Select any crawlers or scrapers you want to block completely from your site.
5. Add custom rules (optional)
Add any additional Disallow or Allow rules. One rule per line. Example: Disallow: /private/
Your robots.txt
What is a robots.txt file and how does it work?
The robots.txt file is a simple text file that tells web crawlers which parts of your site they should and should not access.
Crawl budget management
Blocking low-value URLs like admin pages, search results, and WooCommerce cart pages ensures Google's crawl budget is spent on your important content, not administrative endpoints.
Not a security tool
robots.txt controls which pages crawlers visit, not which pages they can access. A page blocked in robots.txt can still be accessed by anyone with the URL. Do not rely on robots.txt for security.
Critical warning
Never add Disallow: / to your robots.txt on a live production site. This tells all crawlers to skip your entire website, which will remove it from Google's index within days.
robots.txt syntax explained.
User-agent: *
Disallow: /path/
Allow: /path/file
Sitemap: https://...
Crawl-delay: 10
# comment text
Edit your robots.txt without FTP.
Rank Bix SEO includes a built-in robots.txt editor directly in your WordPress admin. No FTP access required. Changes save immediately and apply to your live site.