Zum Inhalt springen
RB

robots.txt Generator

Build robots.txt with User-agent rules, Allow/Disallow, Sitemap and optional Crawl-delay.

Generate a site-root robots.txt. Crawler interpretation varies by engine.

User-agent group 1

one per line
optional

Sitemap

optional / non-standard
# robots.txt generated by toco.tools
# https://example.com

User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

Sitemap: https://example.com/sitemap.xml

Place the output at your site root (https://example.com/robots.txt). Google ignores Crawl-delay. Validate in Search Console (or similar) before relying on it in production.

Über robots.txt Generator

robots.txt lives at your site root and tells crawlers which paths they may fetch. Writing it by hand is easy to get wrong — missing User-agent blocks or invalid Sitemap URLs are common. toco.tools’s robots.txt Generator builds multiple User-agent groups with Allow/Disallow rules, optional Crawl-delay, multiple Sitemaps, and an optional Host line. You get presets (allow all / block all / WordPress-ish), light syntax checks, copy, and download. Crawl-delay is ignored by Googlebot. Output is a draft file, not a guarantee of crawler behavior. Everything runs in your browser; nothing is sent to toco.tools servers.

So funktioniert's

  1. Pick a preset or enter User-agent and Allow/Disallow rules
  2. Optionally add Crawl-delay, Sitemap URLs and Host
  3. Copy or download the robots.txt and place it at your site root

FAQ

Does Crawl-delay work with Google?

No. Googlebot ignores Crawl-delay. Some other crawlers (e.g. Bing) may honor it. Consider Search Console crawl controls for Google.

What does an empty Disallow mean?

A bare `Disallow:` (no path) is the conventional “no restrictions / allow all” form — used by the Allow all preset. To block everything, use `Disallow: /`.

Is my input sent anywhere?

No. Generation runs entirely in your browser; nothing is sent to toco.tools servers.