Robots.txt Generator
Create a robots.txt file with allow and disallow rules per user agent, including presets for AI crawlers like GPTBot, ClaudeBot and PerplexityBot.
- 01Add user agents and rules
- 02Choose AI crawler policy
- 03Add your sitemap URL
- 04Copy or download robots.txt
Blocking a path in robots.txt prevents crawling, not indexing. Use noindex for pages that must stay out of results.
Ignored by Googlebot; respected by some other crawlers.
User-agent: *
Allow: /wp-content/uploads/
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search?
Disallow: /*?sessionid=
Sitemap: https://example.com/sitemap.xmlPlace at https://example.com/robots.txt. Test in Search Console before deploying; a bad rule can deindex a site.
What this tool tells you
Robots.txt controls crawler access. Mistakes can block your entire site; deliberate policy can shape how AI systems use your content.
Go further
Want this handled for you?
Our consultants use this exact check as part of a full programme. Tell us your URL and we will show you what a site-wide view looks like.
Guides that use this tool
The Complete Guide to Technical SEO
Crawling, indexing, rendering, site architecture, performance and structured data explained from first principles.
The technical SEO checklist we run on every new client site
Forty checks across crawl, index, render and performance — the same list our consultants use on day one.
Faceted navigation SEO: which filter combinations deserve to be indexed
A decision framework for e-commerce teams drowning in filter URLs.
Robots.txt Generator FAQs
Ready to see what is holding your organic growth back?
Start with a free audit of any URL, or book a strategy call and we will walk through the full picture: technical, content, authority and AI visibility.