← All tools
Robots.txt Tester
Fetch and test robots.txt rules to see whether a URL is Allowed or Blocked for a selected user-agent.
Privacy Notice — When you click `Fetch`, we send the site URL to our server to proxy-load/robots.txt. The Allowed/Blocked rule evaluation runs in your browser.
About this tool
Fetch and test `robots.txt` rules to see whether a specific URL is Allowed or Blocked for a chosen user-agent. You can also edit the rules to simulate changes before deploying them.
Common use cases
- Confirm that sensitive paths (e.g. admin pages) are properly Disallowed
- Validate Allow/Disallow precedence for specific URLs
- Try “what-if” edits to robots.txt without uploading to production
- The result uses the `robots-parser` matching logic; other crawlers can behave slightly differently.
- Robots.txt controls crawling, not indexing. To fully prevent indexing, use `X-Robots-Tag: noindex` or meta tags.
How to use
- Enter a website URL and click Fetch to load /robots.txt.
- Optional: edit robots.txt in the box to simulate changes.
- Choose a user-agent (e.g. Googlebot or *).
- Enter a full URL to test; Allowed/Blocked appears immediately.
Fetch will send the site URL to our server to proxy-load /robots.txt. Allowed/Blocked evaluation runs in your browser.
Fetch limit: 512KB. Test logic runs in your browser.
You can enter a path such as
/admin; it resolves against the site URL above.Result
Enter a test URL and click `Test` to see whether that URL is allowed or blocked by `robots.txt`.