[{"data":1,"prerenderedAt":163},["ShallowReactive",2],{"tool-content-en-robots-txt-generator":3},{"id":4,"title":5,"body":6,"description":153,"extension":154,"locale":155,"meta":156,"navigation":157,"path":158,"seo":159,"stem":160,"toolId":161,"__hash__":162},"tools\u002Ftools\u002Fen\u002Frobots-txt-generator.md","robots.txt Generator",{"type":7,"value":8,"toc":145},"minimark",[9,14,39,43,72,89,93,127,131],[10,11,13],"h2",{"id":12},"what-this-tool-does","What this tool does",[15,16,17,18,22,23,26,27,30,31,34,35,38],"p",{},"Assemble a ",[19,20,21],"code",{},"robots.txt"," from structured controls instead of typing the syntax from memory. Each crawler group pairs one or more ",[19,24,25],{},"User-agent"," tokens with its own ",[19,28,29],{},"Allow"," and ",[19,32,33],{},"Disallow"," rules and an optional ",[19,36,37],{},"Crawl-delay",", and the file is generated on the right as you edit. Everything runs in your browser, and nothing is uploaded.",[10,40,42],{"id":41},"crawler-groups","Crawler groups",[15,44,45,46,50,51,30,54,57,58,30,61,64,65,68,69,71],{},"A group starts with the crawler tokens it targets. Type any token by hand or open ",[47,48,49],"strong",{},"Preset crawlers"," for the common ones: search engines such as ",[19,52,53],{},"Googlebot",[19,55,56],{},"Bingbot",", AI crawlers such as ",[19,59,60],{},"GPTBot",[19,62,63],{},"ClaudeBot",", SEO tools, and social preview bots. The wildcard ",[19,66,67],{},"*"," matches every robot, so a single ",[19,70,67],{}," group is what most sites start with. A group without a user-agent is skipped, because it would not be valid robots.txt.",[15,73,74,75,77,78,80,81,84,85,88],{},"Rules keep the order you give them, which matters: a more specific ",[19,76,29],{}," should come after the broader ",[19,79,33],{}," it re-opens. An empty ",[19,82,83],{},"Disallow:"," under ",[19,86,87],{},"User-agent: *"," is a valid way to block nothing.",[10,90,92],{"id":91},"sitemaps-host-and-validation","Sitemaps, Host, and validation",[15,94,95,96,99,100,103,104,106,107,109,110,112,113,116,117,119,120,122,123,126],{},"Add one or more ",[47,97,98],{},"Sitemaps"," to advertise them to crawlers, and an optional ",[47,101,102],{},"Host"," for Yandex. As the file grows, the footer checks it for the mistakes people make most often: rules before any ",[19,105,25],{},", an empty or repeated ",[19,108,25],{},", a ",[19,111,25],{}," token containing spaces, rule paths that do not start with ",[19,114,115],{},"\u002F"," or ",[19,118,67],{},", a non-numeric ",[19,121,37],{},", and sitemap URLs without an ",[19,124,125],{},"http(s)"," scheme. Every problem is listed with its line number so you can jump straight to it.",[10,128,130],{"id":129},"copying-and-saving","Copying and saving",[15,132,133,134,137,138,141,142,144],{},"Use ",[47,135,136],{},"Copy"," to put the result on the clipboard, or ",[47,139,140],{},"Download"," to save it as ",[19,143,21],{},". Your groups, rules, sitemaps, and host are remembered on this device when tool memory is enabled.",{"title":146,"searchDepth":147,"depth":147,"links":148},"",2,[149,150,151,152],{"id":12,"depth":147,"text":13},{"id":41,"depth":147,"text":42},{"id":91,"depth":147,"text":92},{"id":129,"depth":147,"text":130},"Build and validate robots.txt rules for any crawler with Allow, Disallow, Crawl-delay, and Sitemap directives, then copy or download the file.","md","en",{},true,"\u002Ftools\u002Fen\u002Frobots-txt-generator",{"title":5,"description":153},"tools\u002Fen\u002Frobots-txt-generator","robots-txt-generator","_NKfNtQBHOy0ufbEo0rk_K-RsicSgVVL_CTC6aqwl9k",1792391420216]