Robots.txt Generator

About the Robots.txt Generator

Most well-behaved crawlers check robots.txt before they touch anything else on a site, which makes it the first line of control over what gets crawled at all. Add a separate User-agent block per bot, list the paths each one should skip or is explicitly allowed to visit, and optionally point to a sitemap, this builds it all into a file ready to drop into a site's root, download button included for saving it directly.

Robots.txt controls crawling, not indexing

This is the single most common misunderstanding about this file: Disallow tells a crawler not to fetch a URL, it doesn't tell Google to keep that URL out of search results. If other pages link to a disallowed URL, Google can still list it in search results as a bare URL with no title or snippet, because it never crawled the page to know what's on it. Keeping a page out of search results entirely requires a noindex meta tag or header on the page itself, which means the page has to be crawlable for Google to see that instruction in the first place, disallowing a page in robots.txt while also relying on noindex to hide it is a contradiction that actually keeps the page indexed.

Where the file has to live

Robots.txt only works when it's placed at the exact root of a domain, https://example.com/robots.txt, not inside a subfolder and not served from a subdomain to cover the main domain. A file at example.com/blog/robots.txt is simply never read by crawlers looking for site-wide rules, and each subdomain that needs its own rules needs its own robots.txt at its own root. A staging subdomain like staging.example.com, for one, needs its own separate file if it's meant to be excluded from crawling, the production domain's robots.txt has no effect on it whatsoever.

It's a request, not a lock

Well-known crawlers like Googlebot and Bingbot respect robots.txt because they're built to, but nothing about the file technically prevents a request, it's an honor-system convention agreed on across the web, not an access control mechanism enforced by anything technical. A scraper that ignores robots.txt entirely can still fetch a disallowed path directly, so anything genuinely sensitive, an admin panel, a staging environment, private user data, needs real authentication rather than a Disallow line as its actual protection.

Multiple blocks and overlapping rules

Separate User-agent blocks let different rules apply to different bots, a stricter set for an aggressive crawler and a looser set for Googlebot, for instance. When Allow and Disallow rules for the same bot overlap on similar paths, Google's crawler follows the more specific, longer matching path rather than whichever rule appears first in the file, so a broad Disallow: /private/ paired with a narrower Allow: /private/pricing/ lets that one page through even though its parent folder is otherwise blocked.

The sitemap line, and what this tool leaves out

The Sitemap directive at the bottom applies to the whole file regardless of which User-agent block it's written near, it's a hint pointing crawlers to the XML sitemap rather than a rule tied to a specific bot, which is why this generator adds it once at the end instead of repeating it per block. One thing intentionally left out here is Crawl-delay, Google has never respected that directive, and while Bing and a few others do, setting it too aggressively can slow down how quickly new or updated pages get discovered, it's a setting worth adding by hand only for a specific bot causing a real server-load problem, not as a default.