Live
No signup
robots.txt Generator
robots.txt without mistakes.
Settings
Start from a preset and customize.
0 = no delay. Some bots ignore this field.
robots.txt
# robots.txt — generado por Zona Utility User-agent: * Allow: / # Custom rules User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /private/ # Bloquear AI crawlers User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: PerplexityBot Disallow: / User-agent: Applebot-Extended Disallow: / Sitemap: https://tusitio.com/sitemap.xml
Upload to your domain root: /robots.txt
How this tool works
robots.txt is the first file any crawler (Googlebot, Bingbot, AhrefsBot, GPTBot) reads when hitting your site. It tells them what to crawl, how fast, and where your sitemap lives. A misconfigured robots.txt can block your entire site from Google without warning.
This generator builds the file from safe presets (WordPress, Next.js, Shopify), lets you add custom rules and validates syntax. It also blocks AI crawlers by default (GPTBot, ClaudeBot, CCBot) if you want to preserve your content.
Frequently asked questions
- Is robots.txt mandatory?
- No, but highly recommended. Without it, crawlers assume full permission and may hit your staging environment.
- Does Disallow block indexing?
- Blocks crawling, not indexing. To keep a URL out of Google, use
noindexin meta or HTTP header. - Block OpenAI / Anthropic?
- Yes, with
User-agent: GPTBot Disallow: /andUser-agent: ClaudeBot Disallow: /. They honor it, but it's not legally binding. - Where do I upload it?
- Domain root:
https://yoursite.com/robots.txt. Not a subpath — Google only reads the root.