offlinewebtools.com — free, watermark-free browser tools that stay on your device with nothing uploading to our servers

Robots.txt Generator

Generate robots.txt files to control search engine crawlers

What this tool does

Robots.txt Generator helps you generate robots.txt files to control search engine crawlers.

What you get

Robots.txt Generator produces a direct output in this page and keeps all processing on your device, with nothing uploaded to our servers.

Free to use, no watermarks, runs in your browser—nothing uploads to our servers

Select which bots these rules apply to (* = all bots)

Common patterns: / (all), /admin/ (admin folder), /*.pdf (PDF files)

Time to wait between requests

User-agent: *
Disallow: /admin/

Tips:

  • Place robots.txt in your website's root directory
  • Use Disallow: / to block all crawlers from your entire site
  • Use Allow to override more restrictive Disallow rules
  • Test your robots.txt with Google Search Console

Jump to helper sections

Jump to FAQs Jump to related tools

Report a bug

If this tool misbehaves, share a quick summary so we can investigate. We include the tool URL automatically—no need to copy and paste.