The most specific robots.txt rule wins, not the first one written

Every site publishes a robots.txt file, the plain-text file at its root that tells automated crawlers which paths they may fetch. When that file has an Allow rule and a Disallow rule that both match the same URL, Google does not read top to bottom. It measures. The rule with the longer path wins, and where the line sits in the file changes nothing.¹ I fetched the robots.txt of 19 large US retailers and found 7 of them shipping an Allow rule nested inside a directory they had already blocked, which is exactly the arrangement where reading in order gives you the wrong answer. Two of those sites write the Disallow first. The Allow still wins.

Across every overlapping rule pair on 19 retail sites, exactly zero were the same length. The famous tie-breaker never fired once.

What happens when an Allow and a Disallow both match a URL?

Google compares the two rules and keeps the more specific one. Its specification states the test plainly: crawlers use the rule with the longer path.¹ When two rules conflict, including ones using a wildcard, a symbol standing in for any text, Google falls back to the less restrictive rule. Length decides first, and that fallback only settles an exact tie.

Two more words before we go on. A crawler is an automated program a search engine uses to fetch pages, and Googlebot is Google’s. A path is everything in a URL after the domain name, so /includes/home is a path, and a path ending in a slash groups everything beneath it. Nothing in Google’s specification refers to the order the lines appear in.

How does Google measure “more specific”?

Google counts the characters in the rule path, wildcards included. A rule reading Disallow: /includes/ has a path of 10 characters. A rule reading Allow: /includes/home/homeBodyContent has 30. The second is more specific, so it wins for any URL both of them match, and the first one still applies everywhere else under that directory.

The specification demonstrates this with six worked examples, and one of them is worth knowing because it breaks the simple version of the rule. For the URL https://example.com/page.htm, with allow: /$ and disallow: /, the winner is the Disallow.¹ The $ anchors the Allow to the end of the URL, so it only ever matches the root. A rule that does not match never enters the comparison at all. Indexed, throughout this piece, means a page Google has stored and can show in results.

Do real sites actually ship this conflict?

Yes, and more often than I expected: nine of 19 large US retailers carry the conflict right now. On September 1, 2026 I fetched /robots.txt from the 19 US retail sites already validated for the crawl budget piece. All 19 answered. Nine of them carry at least one pair where an Allow and a Disallow could both match a single URL inside the group Google itself would obey, 180 such pairs.

Counting every group instead of Google’s raises that to 11 sites and 491 pairs. The gap between those two counts is itself the group gate described below, and I got it wrong on the first pass. My original count included a carve-out on Nike that lives entirely under Chinese search-engine user agents, where Googlebot never looks. Splitting the pairs by which rule wins matters too, because the two directions are different situations.

What does a carve-out look like on a live site?

Seven of the 19 retail sites I checked ship what I will call a specificity carve-out, an Allow written deeper than a Disallow that already covers it, re-opening one path inside a blocked directory. Walgreens is the cleanest example, and it is two lines long.

Disallow: /includes/
Allow: /includes/home/homeBodyContent

Read in order, the block comes first and the URL looks blocked. Measured, the Allow is 30 characters against 10, so Google fetches it. Home Depot does the same thing with a wildcard, blocking /s/ and then re-opening /s/*?CI-5. Lowe’s, IKEA, REI, Etsy and Wayfair all carry the same shape.

Two sites, CVS and Gap, have pairs running only the other way, where the Disallow is the longer rule. That is the ordinary arrangement of a broad Allow with specific blocks underneath, and it surprises nobody. Gap is worth one note, because its pairs sit in a group it wrote specifically for Googlebot rather than in the wildcard group.

What about the tie-breaker everyone quotes?

The least-restrictive tie-breaker never fired once on this 19-site panel. Across every overlapping pair, the 180 in Google’s groups and the 491 counting all groups alike, not a single pair had an Allow and a Disallow of exactly equal path length, so the clause had no occasion to decide anything. It is also the half of the rule I reach for first when explaining precedence, which is the imbalance worth noticing.

One null result on one panel is worth holding lightly, since it covers one industry on one day. The point is about proportion. Length settles almost everything on this panel, so remembering only “Allow wins ties” means holding the rare branch and skipping the common one.

Which rules is Google even comparing?

Google compares only the rules inside a single user-agent group, and it picks that group before comparing anything. Its specification says crawlers find the group whose user-agent line is the most specific match for their own name, and that all other groups are ignored. A user-agent is the name a crawler announces itself with, like Googlebot.¹ A group is one or more User-agent: lines followed by the rules that belong to them.

Choosing the group first is a group gate, a second specificity contest sitting above the first and settled before any rule is compared. It runs in the opposite direction to most people’s expectations about defaults. If your file has a User-agent: * group and a User-agent: Googlebot group, Googlebot reads the second one and never sees the first. Rules you assumed applied to everyone do not apply to the crawler you most cared about.

The failure mode is a Disallow you added to the wildcard group for safety, sitting in a file that also names Googlebot somewhere above it. Nothing warns you. The rule is simply never evaluated for that crawler, and a URL you believe is blocked gets fetched on the next pass.

Why does “the first rule wins” feel right?

The habit comes from almost everything else we read top to bottom. A form’s later answer overrides an earlier one, and a contract’s specific exception beats its general clause. Robots.txt looks like the same kind of list, so the habit transfers. The analogy breaks at the one place that matters: in a contract the specific clause wins because a human read both and judged, whereas here it wins because a machine counted characters.

The consequence is quiet, which is why it survives. A carve-out that you believe is blocked simply gets crawled, and nothing anywhere reports a problem, because from Google’s side nothing went wrong.

How do wildcards change the match?

Two characters do the work in a robots.txt path, and both of them affect precedence. An asterisk matches any run of characters, so the pattern /s/*?CI-5 matches the real URL /s/drill?CI-5. A dollar sign anchors the pattern to the end of the URL, so /$ matches the site root and nothing below it. Every other character is matched literally.¹

Because both symbols count toward that length, a short pattern with a wildcard can beat a longer literal one. An anchored pattern can also lose to a shorter rule simply by never matching the URL at all. Match first, then measure.

How is this different from the robots.txt tester already on this site?

The AI robots.txt tester answers a different question. It takes a URL and a roster of crawlers and tells you which of them are allowed to fetch it, which is what you want when you are auditing access for a specific bot.

The tool below answers the question underneath that one. Given two rules that both match, which one applies, and why. You need the second answer when a file contains a carve-out and you are trying to work out whether it does what its author intended.

How do you check your own file?

Take any URL you believe is blocked and find every Allow and Disallow line in the matching user-agent group whose path is a prefix of it. If only one matches, that is your answer. If two match, count the characters in each rule path and keep the longer one, remembering that a rule ending in $ only matches when the URL ends there.

Then check the direction. A Disallow that is longer than its Allow is ordinary housekeeping. An Allow that is longer than its Disallow is a carve-out, and a carve-out is a decision somebody made on purpose. Confirm it was you, because a blocked URL can still be indexed from external links, which is the ground the robots piece covers, and blocked URLs show up in Search Console, Google’s free reporting tool for your own site, under “URL blocked by robots.txt” whether the block was intended or not.²