offlinewebtools.com browser-local workbench

Robots.txt Generator

Generate robots.txt files to control search engine crawlers

Shipped
Browser local

Select which bots these rules apply to (* = all bots)

Common patterns: / (all), /admin/ (admin folder), /*.pdf (PDF files)

Time to wait between requests

User-agent: *
Disallow: /admin/

Tips:

  • Place robots.txt in your website's root directory
  • Use Disallow: / to block all crawlers from your entire site
  • Use Allow to override more restrictive Disallow rules
  • Test your robots.txt with Google Search Console

What this tool does

Robots.txt Generator helps you generate robots.txt files to control search engine crawlers.

What you get

Robots.txt Generator produces a direct output in this page and keeps all processing on your device, with nothing uploaded to our servers.

Jump to helper sections

Jump to FAQs Jump to related tools

Report a bug

If this tool misbehaves, share a quick summary so we can investigate. We include the tool URL automatically—no need to copy and paste.