Aller au contenu
RB

robots.txt Generator

Build robots.txt with User-agent rules, Allow/Disallow, Sitemap and optional Crawl-delay.

Generate a site-root robots.txt. Crawler interpretation varies by engine.

User-agent group 1

one per line
optional

Sitemap

optional / non-standard
# robots.txt generated by toco.tools
# https://example.com

User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

Sitemap: https://example.com/sitemap.xml

Place the output at your site root (https://example.com/robots.txt). Google ignores Crawl-delay. Validate in Search Console (or similar) before relying on it in production.

À propos de robots.txt Generator

robots.txt lives at your site root and tells crawlers which paths they may fetch. Writing it by hand is easy to get wrong — missing User-agent blocks or invalid Sitemap URLs are common. toco.tools’s robots.txt Generator builds multiple User-agent groups with Allow/Disallow rules, optional Crawl-delay, multiple Sitemaps, and an optional Host line. You get presets (allow all / block all / WordPress-ish), light syntax checks, copy, and download. Crawl-delay is ignored by Googlebot. Output is a draft file, not a guarantee of crawler behavior. Everything runs in your browser; nothing is sent to toco.tools servers.

Comment l'utiliser

  1. Pick a preset or enter User-agent and Allow/Disallow rules
  2. Optionally add Crawl-delay, Sitemap URLs and Host
  3. Copy or download the robots.txt and place it at your site root

FAQ

Does Crawl-delay work with Google?

No. Googlebot ignores Crawl-delay. Some other crawlers (e.g. Bing) may honor it. Consider Search Console crawl controls for Google.

What does an empty Disallow mean?

A bare `Disallow:` (no path) is the conventional “no restrictions / allow all” form — used by the Allow all preset. To block everything, use `Disallow: /`.

Is my input sent anywhere?

No. Generation runs entirely in your browser; nothing is sent to toco.tools servers.