Robots.txt Generator & Validator
Build a robots.txt file with rules for multiple user agents and sitemap declarations, then run the validator to catch mistakes before you deploy.
Rules
Rule #1
Rule #2
Sitemaps
robots.txt
Validation
No issues detected. Save this file to the root of your site and monitor Search Console.
Quick reference
User-agent: *matches every crawler.Allow: /+ targetedDisallow:is the safest pattern.- Never disallow your own sitemap or CSS/JS files — crawlers need them.
- Paths are matched by prefix, not wildcards (except some engines).
- The most specific rule wins when engines support precedence.
- Test with Google’s robots.txt Tester or Search Console after deploying.
Frequently asked questions
Does robots.txt hide my pages from users?
No. Browsers and visitors ignore robots.txt — it only guides cooperating crawlers. To hide content from visitors, use authentication or a noindex directive instead.
Where do I put the file?
At the web root, e.g. https://example.com/robots.txt. It is fetched before other pages, so keep it small and fast.
What about the Sitemap directive?
Not every engine supports it, but Google and Bing do — it is a zero-cost way to point crawlers at your sitemap.
Related tools
Related tools
More from SEO & Marketing
Get a free SEO site audit
We review your pages and tell you exactly what to fix to rank higher.
Was this tool useful?