Robots.txt Tester
Paste your robots.txt and check whether a given path is allowed or blocked for a specific crawler, updates live as you type.
About the Robots.txt Tester
Under the hood this applies the same longest-match precedence real crawlers use: within the matched User-agent group, every Allow and Disallow rule is tested against a path, and whichever rule covers the most characters wins, with ties resolved in favor of Allow. The * wildcard and $ end-of-string anchor behave exactly as the robots.txt spec defines them, so the verdict here should line up with how a real crawler would treat the same file.
Which User-agent group actually applies
A robots.txt file with a block for User-agent: Googlebot and a separate block for User-agent: * doesn't merge the two, a crawler that matches the named block uses only that block's rules, the wildcard block is ignored entirely for that bot. This trips people up constantly, a Disallow rule written only in the * block is assumed to apply to every crawler, but a bot with its own named block skips right past it. Testing the same path against both * and a specific bot name here, like Googlebot or Bingbot, is the fastest way to catch that mismatch before it becomes a real crawling surprise.
How ties between Allow and Disallow are broken
When an Allow rule and a Disallow rule match a path with the exact same length, this tool sides with Allow, matching the tie-breaking behavior described in Google's own robots.txt documentation. Outside of an exact tie, length alone decides it, a Disallow rule that's more specific (longer) than a matching Allow rule wins even if the Allow rule was written first or looks like it should take priority by intent.
Reading the wildcard and anchor characters
* inside a path matches any sequence of characters, so /private/*.pdf matches any PDF nested anywhere under /private/. A path with no wildcard is still matched as a prefix, Disallow: /admin blocks /admin, /admin/, and /admin-panel alike, which is a common source of accidentally over-broad rules. Adding a trailing $, as in Disallow: /admin$, anchors the match to the end of the string, blocking only that exact path and leaving everything nested under it untouched.
When no rule matches at all
Robots.txt defaults to allow, if no Disallow rule in the applicable group matches a path, that path is crawlable, there's no implicit deny-by-default behavior to account for. An empty file, a missing file, or a file with no group matching the tested User-agent all resolve to the same outcome here, allowed, which mirrors how real crawlers treat those same situations.
Why testing beats reading the file by eye
A robots.txt file with more than a handful of rules is genuinely hard to reason about by reading it top to bottom, the actual outcome for a given path depends on which User-agent group gets selected, then which single rule out of potentially a dozen ends up being the longest match, not on the order the lines happen to be written in. Pasting the file in here and testing a handful of real paths, including ones that are only supposed to be partially blocked, catches mismatches between what a robots.txt file was intended to do and what it actually does that are easy to miss just by scanning the text.
This is also the natural next step after building a file with the Robots.txt Generator on this site, generating the rules and then testing a few representative paths against a couple of different User-agent values confirms the file behaves as intended before it goes live on a real domain, catching a mistake here costs a few seconds, catching it after a search engine has already stopped crawling an important section of a site costs a lot more.
Frequently Asked Questions
Does a rule in the * block apply to every crawler?
No, only to crawlers that don't have their own named User-agent block. If a file has a separate block for a specific bot like Googlebot, that bot uses only its own block's rules and ignores the wildcard block entirely, even for paths the wildcard block covers.
What happens when an Allow and a Disallow rule both match the same path?
The rule matching more characters of the path wins. If both rules match exactly the same length, Allow wins the tie, matching the precedence Google's own documentation describes.
What does the $ symbol do in a robots.txt rule?
It anchors the rule to the end of the path, so it matches only that exact path rather than everything nested underneath it. Without it, a rule like Disallow: /admin also blocks /admin/anything and /admin-panel as a prefix match.
Is a path allowed if robots.txt has no matching rule?
Yes. Robots.txt defaults to allowing crawling, a path is only blocked if a Disallow rule in the applicable User-agent group actually matches it. A missing file, an empty file, or no matching group all default to allowed.