robots.txt Generator

What this tool does

Assemble a robots.txt from structured controls instead of typing the syntax from memory. Each crawler group pairs one or more User-agent tokens with its own Allow and Disallow rules and an optional Crawl-delay, and the file is generated on the right as you edit. Everything runs in your browser, and nothing is uploaded.

Crawler groups

A group starts with the crawler tokens it targets. Type any token by hand or open Preset crawlers for the common ones: search engines such as Googlebot and Bingbot, AI crawlers such as GPTBot and ClaudeBot, SEO tools, and social preview bots. The wildcard * matches every robot, so a single * group is what most sites start with. A group without a user-agent is skipped, because it would not be valid robots.txt.

Rules keep the order you give them, which matters: a more specific Allow should come after the broader Disallow it re-opens. An empty Disallow: under User-agent: * is a valid way to block nothing.

Sitemaps, Host, and validation

Add one or more Sitemaps to advertise them to crawlers, and an optional Host for Yandex. As the file grows, the footer checks it for the mistakes people make most often: rules before any User-agent, an empty or repeated User-agent, a User-agent token containing spaces, rule paths that do not start with / or *, a non-numeric Crawl-delay, and sitemap URLs without an http(s) scheme. Every problem is listed with its line number so you can jump straight to it.

Copying and saving

Use Copy to put the result on the clipboard, or Download to save it as robots.txt. Your groups, rules, sitemaps, and host are remembered on this device when tool memory is enabled.